Learning particle swarming models from data with Gaussian processes
HTML articles powered by AMS MathViewer
- by Jinchao Feng, Charles Kulick, Yunxiang Ren and Sui Tang;
- Math. Comp. 93 (2024), 2391-2437
- DOI: https://doi.org/10.1090/mcom/3915
- Published electronically: November 15, 2023
- HTML | PDF | Request permission
Abstract:
Interacting particle or agent systems that exhibit diverse swarming behaviors are prevalent in science and engineering. Developing effective differential equation models to understand the connection between individual interaction rules and swarming is a fundamental and challenging goal. In this paper, we study the data-driven discovery of a second-order particle swarming model that describes the evolution of $N$ particles in $\mathbb {R}^d$ under radial interactions. We propose a learning approach that models the latent radial interaction function as Gaussian processes, which can simultaneously fulfill two inference goals: one is the nonparametric inference of the interaction function with pointwise uncertainty quantification, and the other is the inference of unknown scalar parameters in the noncollective friction forces of the system. We formulate the learning problem as a statistical inverse learning problem and introduce an operator-theoretic framework that provides a detailed analysis of recoverability conditions, establishing that a coercivity condition is sufficient for recoverability. Given data collected from $M$ i.i.d trajectories with independent Gaussian observational noise, we provide a finite-sample analysis, showing that our posterior mean estimator converges in a Reproducing Kernel Hilbert Space norm, at an optimal rate in $M$ equal to the one in the classical 1-dimensional Kernel Ridge regression. As a byproduct, we show we can obtain a parametric learning rate in $M$ for the posterior marginal variance using $L^{\infty }$ norm and that the rate could also involve $N$ and $L$ (the number of observation time instances for each trajectory) depending on the condition number of the inverse problem. We provide numerical results on systems exhibiting different swarming behaviors, highlighting the effectiveness of our approach in the scarce, noisy trajectory data regime.References
- W. F. Ames, and B. Pachpatte, Inequalities for Differential and Integral Equations, Elsevier, vol. 197 (1997).
- C. Archambeau, D. Cornford, M. Opper, and J. Shawe-Taylor, Gaussian process approximations of stochastic differential equations, Gaussian Processes in Practice, pp. 1–16, PMLR (2007).
- Frank Bauer, Sergei Pereverzev, and Lorenzo Rosasco, On regularization algorithms in learning theory, J. Complexity 23 (2007), no. 1, 52–72. MR 2297015, DOI 10.1016/j.jco.2006.07.001
- Fabian Baumann, Igor M. Sokolov, and Melvyn Tyloo, A Laplacian approach to stubborn agents and their role in opinion formation on influence networks, Phys. A 557 (2020), 124869, 19. MR 4119455, DOI 10.1016/j.physa.2020.124869
- Rajendra Bhatia, Matrix analysis, Graduate Texts in Mathematics, vol. 169, Springer-Verlag, New York, 1997. MR 1477662, DOI 10.1007/978-1-4612-0653-8
- Jaya Prakash Narayan Bishwal, Estimation in interacting diffusions: continuous and discrete sampling, Appl. Math. (Irvine) 2 (2011), no. 9, 1154–1158. MR 2924983, DOI 10.4236/am.2011.29160
- Gilles Blanchard and Nicole Mücke, Optimal rates for regularization of statistical inverse learning problems, Found. Comput. Math. 18 (2018), no. 4, 971–1013. MR 3833647, DOI 10.1007/s10208-017-9359-7
- Jiří Blank, Pavel Exner, and Miloslav Havlíček, Hilbert space operators in quantum physics, 2nd ed., Theoretical and Mathematical Physics, Springer, New York; AIP Press, New York, 2008. MR 2458485
- Mattia Bongini, Massimo Fornasier, Markus Hansen, and Mauro Maggioni, Inferring interaction rules from observations of evolutive systems I: the variational approach, Math. Models Methods Appl. Sci. 27 (2017), no. 5, 909–951. MR 3636616, DOI 10.1142/S0218202517500208
- Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci. USA 113 (2016), no. 15, 3932–3937. MR 3494081, DOI 10.1073/pnas.1517384113
- A. Caponnetto, and E. De Vito, Fast rates for regularized least-squares algorithm, Technical Report, MIT (2005).
- Jiuhai Chen, Lulu Kang, and Guang Lin, Gaussian process assisted active learning of physical laws, Technometrics 63 (2021), no. 3, 329–342. MR 4296900, DOI 10.1080/00401706.2020.1817790
- Xiaohui Chen, Maximum likelihood estimation of potential energy in interacting particle systems from single-trajectory data, Electron. Commun. Probab. 26 (2021), Paper No. 45, 13. MR 4284626, DOI 10.1214/21-ecp416
- Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M. Stuart, Solving and learning nonlinear PDEs with Gaussian processes, J. Comput. Phys. 447 (2021), Paper No. 110668, 29. MR 4311012, DOI 10.1016/j.jcp.2021.110668
- Yao-li Chuang, Maria R. D’Orsogna, Daniel Marthaler, Andrea L. Bertozzi, and Lincoln S. Chayes, State transitions and the continuum limit for a 2D interacting, self-propelled particle system, Phys. D 232 (2007), no. 1, 33–47. MR 2369988, DOI 10.1016/j.physd.2007.05.007
- D. A. Cohn, Z. Ghahramani, and M. I. Jordan, Active learning with statistical models, J. Artificial Intell. Res. 4 (1996) 129–145.
- Felipe Cucker and Steve Smale, On the mathematical foundations of learning, Bull. Amer. Math. Soc. (N.S.) 39 (2002), no. 1, 1–49. MR 1864085, DOI 10.1090/S0273-0979-01-00923-5
- Ernesto De Vito, Lorenzo Rosasco, Andrea Caponnetto, Umberto De Giovannini, and Francesca Odone, Learning from examples as an inverse problem, J. Mach. Learn. Res. 6 (2005), 883–904. MR 2249842
- L. Della Maestra and M. Hoffmann, The lan property for mckean-vlasov models in a mean-field regime, Preprint, arXiv:2205.05932, 2022.
- L. Devroye, L. Györfi, and G. Lugosi, A probabilistic theory of pattern recognition, Springer Science & Business Media, vol. 31, (2013).
- M. R. D’Orsogna, Y.-L. Chuang, A. L. Bertozzi, and L. S. Chayes, Self-propelled particles with soft-core interactions: patterns, stability, and collapse, Physical review letters, 96(10):104302, (2006).
- M. H. H. Engle, and A. Neubauer, Regularization of Inverse Problems, Mathematics and its Applications, vol. 375 (1996).
- V. Genon-Catalot, and C. Larédo, Inference for ergodic McKean-Vlasov stochastic differential equations with polynomial interactions, hal-03866218v2, (2022).
- Susana N. Gomes, Andrew M. Stuart, and Marie-Therese Wolfram, Parameter estimation for macroscopic pedestrian dynamics models from microscopic data, SIAM J. Appl. Math. 79 (2019), no. 4, 1475–1500. MR 3989251, DOI 10.1137/18M1215980
- Robert B. Gramacy and Daniel W. Apley, Local Gaussian process approximation for large computer experiments, J. Comput. Graph. Statist. 24 (2015), no. 2, 561–578. MR 3357395, DOI 10.1080/10618600.2014.914442
- M. Gu, X. Liu, X. Fang, and S. Tang, Scalable marginalization of latent variables for correlated data, Preprint, arXiv:2203.08389, 2022.
- L. Györfi, M. Kohler, A. Krzyzak, and H. Walk, A Distribution-Free Theory of Nonparametric Regression, Springer Science & Business Media (2006).
- M. Heinonen, C. Yildiz, H. Mannerström, J. Intosalmi, and H. Lähdesmäki, Learning unknown ODE models with gaussian processes, International Conference on Machine Learning, pp. 1959–1968, PMLR (2018).
- M. Kanagawa, P. Hennig, D. Sejdinovic, and B. K. Sriperumbudur, Gaussian processes and kernel methods: A review on connections and equivalences, Preprint, arXiv:1807.02582, 2018.
- Raphael A. Kasonga, Maximum likelihood theory for large interacting systems, SIAM J. Appl. Math. 50 (1990), no. 3, 865–875. MR 1050917, DOI 10.1137/0150050
- Matthias Katzfuss and Joseph Guinness, A general framework for Vecchia approximations of Gaussian processes, Statist. Sci. 36 (2021), no. 1, 124–141. MR 4194207, DOI 10.1214/19-STS755
- J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein, Deep neural networks as gaussian processes, Preprint, arXiv:1711.00165, 2017.
- Seungjoon Lee, Mahdi Kooshkbaghi, Konstantinos Spiliotis, Constantinos I. Siettos, and Ioannis G. Kevrekidis, Coarse-scale PDEs from fine-scale observations via machine learning, Chaos 30 (2020), no. 1, 013141, 14. MR 4054767, DOI 10.1063/1.5126869
- Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar, Physics-informed neural operator for learning partial differential equations, Preprint, arXiv:2111.03794, 2021.
- Dong C. Liu and Jorge Nocedal, On the limited memory BFGS method for large scale optimization, Math. Programming 45 (1989), no. 3, (Ser. B), 503–528. MR 1038245, DOI 10.1007/BF01589116
- Q. Liu, Stein variational gradient descent as gradient flow, Advances in Neural Information Processing Systems, vol. 30, Curran Associates, Inc., (2017).
- Z. Long, Y. Lu, X. Ma, and B. Dong, PDE-Net: learning PDEs from data, International Conference on Machine Learning, pp. 3208–3216, PMLR (2018).
- F. Lu, M. Maggioni, and S. Tang, Learning interaction kernels in stochastic systems of interacting particles from multiple trajectories, Preprint, arXiv:2007.15174, 2020.
- F. Lu, M. Maggioni, and S. Tang, Learning interaction kernels in heterogeneous systems of agents from multiple trajectories, Journal of Machine Learning Research, 22(32):1–67, (2021).
- Fei Lu, Ming Zhong, Sui Tang, and Mauro Maggioni, Nonparametric inference of interaction laws in systems of agents from trajectory data, Proc. Natl. Acad. Sci. USA 116 (2019), no. 29, 14424–14433. MR 3984488, DOI 10.1073/pnas.1822012116
- Zhiping Mao, Zhen Li, and George Em Karniadakis, Nonlocal flocking dynamics: learning the fractional order of PDEs from particle simulations, Commun. Appl. Math. Comput. 1 (2019), no. 4, 597–619. MR 4022347, DOI 10.1007/s42967-019-00031-y
- Song Mei, Andrea Montanari, and Phan-Minh Nguyen, A mean field view of the landscape of two-layer neural networks, Proc. Natl. Acad. Sci. USA 115 (2018), no. 33, E7665–E7671. MR 3845070, DOI 10.1073/pnas.1806579115
- Daniel A. Messenger and David M. Bortz, Learning mean-field equations from particle data using WSINDy, Phys. D 439 (2022), Paper No. 133406, 18. MR 4449604, DOI 10.1016/j.physd.2022.133406
- D. A. Messenger, G. E. Wheeler, X. Liu, and D. M. Bortz, Learning anisotropic interaction rules from individual trajectories in a heterogeneous cellular population, Preprint, arXiv:2204.14141, 2022.
- Jason Miller, Sui Tang, Ming Zhong, and Mauro Maggioni, Learning theory for inferring interaction kernels in second-order interacting agent systems, Sampl. Theory Signal Process. Data Anal. 21 (2023), no. 1, Paper No. 21, 58. MR 4604410, DOI 10.1007/s43670-023-00055-9
- Sebastien Motsch and Eitan Tadmor, Heterophilious dynamics enhances consensus, SIAM Rev. 56 (2014), no. 4, 577–621. MR 3274797, DOI 10.1137/120901866
- R. M. Neal, and R. M. Neal, Priors for infinite networks, Bayesian Learning for Neural Networks, pp. 29–53 (1996).
- Tong Qin, Kailiang Wu, and Dongbin Xiu, Data driven governing equations approximation using deep neural networks, J. Comput. Phys. 395 (2019), 620–635. MR 3975726, DOI 10.1016/j.jcp.2019.06.042
- Joaquin Quiñonero-Candela and Carl Edward Rasmussen, A unifying view of sparse approximate Gaussian process regression, J. Mach. Learn. Res. 6 (2005), 1939–1959. MR 2249877
- Maziar Raissi, Deep hidden physics models: deep learning of nonlinear partial differential equations, J. Mach. Learn. Res. 19 (2018), Paper No. 25, 24. MR 3862432
- M. Raissi, P. Perdikaris, and G. Karniadakis, Multistep neural networks for data-driven discovery of nonlinear dynamical systems, Preprint, arXiv:1801.01236, (2018).
- Maziar Raissi, Paris Perdikaris, and George Em Karniadakis, Machine learning of linear differential equations using Gaussian processes, J. Comput. Phys. 348 (2017), 683–693. MR 3689653, DOI 10.1016/j.jcp.2017.07.050
- Mark Rudelson and Roman Vershynin, Hanson-Wright inequality and sub-Gaussian concentration, Electron. Commun. Probab. 18 (2013), no. 82, 9. MR 3125258, DOI 10.1214/ECP.v18-2865
- Håvard Rue, Sara Martino, and Nicolas Chopin, Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations, J. R. Stat. Soc. Ser. B Stat. Methodol. 71 (2009), no. 2, 319–392. MR 2649602, DOI 10.1111/j.1467-9868.2008.00700.x
- Florian Schäfer, Matthias Katzfuss, and Houman Owhadi, Sparse Cholesky factorization by Kullback-Leibler minimization, SIAM J. Sci. Comput. 43 (2021), no. 3, A2019–A2046. MR 4267493, DOI 10.1137/20M1336254
- Louis Sharrock, Nikolas Kantas, Panos Parpas, and Grigorios A. Pavliotis, Online parameter estimation for the McKean-Vlasov stochastic differential equation, Stochastic Process. Appl. 162 (2023), 481–546. MR 4597535, DOI 10.1016/j.spa.2023.05.002
- Steve Smale and Ding-Xuan Zhou, Learning theory estimates via integral operators and their approximations, Constr. Approx. 26 (2007), no. 2, 153–172. MR 2327597, DOI 10.1007/s00365-006-0659-y
- E. Snelson and Z. Ghahramani, Sparse gaussian processes using pseudo-inputs, Adv. Neural Info. Process. Syst. 18 1257–1264, (2005).
- George Stepaniants, Learning partial differential equations in reproducing kernel Hilbert spaces, J. Mach. Learn. Res. 24 (2023), Paper No. [86], 72. MR 4582508
- Jonathan R. Stroud, Michael L. Stein, and Shaun Lysen, Bayesian and maximum likelihood estimation for Gaussian processes on an incomplete lattice, J. Comput. Graph. Statist. 26 (2017), no. 1, 108–120. MR 3610412, DOI 10.1080/10618600.2016.1152970
- Wenpin Tang, Lu Zhang, and Sudipto Banerjee, On identifiability and consistency of the nugget in Gaussian spatial process models, J. R. Stat. Soc. Ser. B. Stat. Methodol. 83 (2021), no. 5, 1044–1070. MR 4349127, DOI 10.1111/rssb.12472
- M. Taylor, Towards a mathematical theory of influence and attitude change, Hum. Relations 21, no. 2, pp. 121–139 (1968).
- Hongqiao Wang and Xiang Zhou, Explicit estimation of derivatives from data and differential equations by Gaussian process regression, Int. J. Uncertain. Quantif. 11 (2021), no. 4, 41–57. MR 4306422, DOI 10.1615/Int.J.UncertaintyQuantification.2021034382
- S. Wang, H. Wang, and P. Perdikaris, Learning the solution operator of parametric partial differential equations with physics-informed deeponets, Preprint, arXiv:2103.10974, 2021.
- Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian processes for machine learning, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2006. MR 2514435
- Shihao Yang, Samuel W. K. Wong, and S. C. Kou, Inference of dynamic systems from noisy and sparse data via manifold-constrained Gaussian processes, Proc. Natl. Acad. Sci. USA 118 (2021), no. 15, Paper No. 2020397118, 8. MR 4294059, DOI 10.1073/pnas.2020397118
- R. Yao, X. Chen, and Y. Yang, Mean-field nonparametric estimation of interacting particle systems, Preprint, arXiv:2205.07937, 2022.
- C. Yildiz, M. Heinonen, J. Intosalmi, H. Mannerstrom, and H. Lahdesmaki, Learning stochastic differential equations with gaussian processes without gradient matching, 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), IEEE, pp. 1–6, (2018).
- Vadim Yurinsky, Sums and Gaussian vectors, Lecture Notes in Mathematics, vol. 1617, Springer-Verlag, Berlin, 1995. MR 1442713, DOI 10.1007/BFb0092599
- Hao Zhang, Inconsistent estimation and asymptotically equal interpolations in model-based geostatistics, J. Amer. Statist. Assoc. 99 (2004), no. 465, 250–261. MR 2054303, DOI 10.1198/016214504000000241
- Z. Zhao, F. Tronarp, R. Hostettler, and S. Särkkä, State-space Gaussian process for drift estimation in stochastic differential equations, ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 5295–5299, (2020).
- Ming Zhong, Jason Miller, and Mauro Maggioni, Data-driven discovery of emergent behaviors in collective dynamics, Phys. D 411 (2020), 132542, 25. MR 4108454, DOI 10.1016/j.physd.2020.132542
Bibliographic Information
- Jinchao Feng
- Affiliation: Department of Mathematics, School of Sciences, Great Bay University, Dongguan, Guangdong 523000, People’s Republic of China
- MR Author ID: 1531758
- ORCID: 0000-0001-9330-4983
- Email: jcfeng@gbu.edu.cn
- Charles Kulick
- Affiliation: Departments of Mathematics, University of California, Santa Barbara, Isla Vista, CA 93107, USA
- MR Author ID: 1556839
- ORCID: 0000-0002-2623-9349
- Email: charles@math.ucsb.edu
- Yunxiang Ren
- Affiliation: Department of Physics, Harvard University, Cambridge, MA 02138, USA
- MR Author ID: 1250599
- Email: yren@g.harvard.edu
- Sui Tang
- Affiliation: Departments of Mathematics, University of California, Santa Barbara, Isla Vista, CA 93107, USA
- MR Author ID: 1107641
- ORCID: 0000-0003-3284-5123
- Email: suitang@ucsb.edu
- Received by editor(s): June 8, 2022
- Received by editor(s) in revised form: June 12, 2022, March 6, 2023, and August 16, 2023
- Published electronically: November 15, 2023
- Additional Notes: The second author and fourth author were partially supported by Faculty Early Career Development Award sponsored by University of California Santa Barbara, Hellman Family Faculty Fellowship, and the NSF DMS-2111303. The second author was partly supported by NSF grant NSF DMS-2111303.
The fourth author is the corresponding author. The authors are listed in alphabetical order. - © Copyright 2023 American Mathematical Society
- Journal: Math. Comp. 93 (2024), 2391-2437
- MSC (2020): Primary 62G05, 70F17, 65F22, 68Q32, 65Cxx; Secondary 70F40, 65Gxx
- DOI: https://doi.org/10.1090/mcom/3915
- MathSciNet review: 4759379