A framework for machine learning of model error in dynamical systems

Levine, Matthew; Stuart, Andrew

doi:10.1090/cams/10

A framework for machine learning of model error in dynamical systems
HTML articles powered by AMS MathViewer

by Matthew E. Levine and Andrew M. Stuart

Comm. Amer. Math. Soc. 2 (2022), 283-344

DOI: https://doi.org/10.1090/cams/10

Published electronically: October 14, 2022

HTML | PDF

Abstract:

The development of data-informed predictive models for dynamical systems is of widespread interest in many disciplines. We present a unifying framework for blending mechanistic and machine-learning approaches to identify dynamical systems from noisily and partially observed data. We compare pure data-driven learning with hybrid models which incorporate imperfect domain knowledge, referring to the discrepancy between an assumed truth model and the imperfect mechanistic model as model error. Our formulation is agnostic to the chosen machine learning model, is presented in both continuous- and discrete-time settings, and is compatible both with model errors that exhibit substantial memory and errors that are memoryless.

First, we study memoryless linear (w.r.t. parametric-dependence) model error from a learning theory perspective, defining excess risk and generalization error. For ergodic continuous-time systems, we prove that both excess risk and generalization error are bounded above by terms that diminish with the square-root of $T$, the time-interval over which training data is specified.

Secondly, we study scenarios that benefit from modeling with memory, proving universal approximation theorems for two classes of continuous-time recurrent neural networks (RNNs): both can learn memory-dependent model error, assuming that it is governed by a finite-dimensional hidden variable and that, together, the observed and hidden variables form a continuous-time Markovian system. In addition, we connect one class of RNNs to reservoir computing, thereby relating learning of memory-dependent error to recent work on supervised learning between Banach spaces using random features.

Numerical results are presented (Lorenz ’63, Lorenz ’96 Multiscale systems) to compare purely data-driven and hybrid approaches, finding hybrid methods less datahungry and more parametrically efficient. We also find that, while a continuous-time framing allows for robustness to irregular sampling and desirable domain- interpretability, a discrete-time framing can provide similar or better predictive performance, especially when data are undersampled and the vector field defining the true dynamics cannot be identified. Finally, we demonstrate numerically how data assimilation can be leveraged to learn hidden dynamics from noisy, partially-observed data, and illustrate challenges in representing memory by this approach, and in the training of such models.

References

Romeo Alexander and Dimitrios Giannakis, Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques, Phys. D 409 (2020), 132520, 24. MR 4093838, DOI 10.1016/j.physd.2020.132520
Ranjan Anantharaman, Yingbo Ma, Shashi Gowda, Chris Laughman, Viral Shah, Alan Edelman, and Chris Rackauckas, Accelerating simulation of stiff nonlinear systems using continuous-time echo state networks, https://arxiv.org/abs/2010.04004v6, 2020.
Mark Asch, Marc Bocquet, and Maëlle Nodet, Data assimilation, Fundamentals of Algorithms, vol. 11, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2016. Methods, algorithms, and applications. MR 3602006, DOI 10.1137/1.9781611974546.pt1
Ibrahim Ayed, Emmanuel de Bézenac, Arthur Pajot, Julien Brajard, and Patrick Gallinari, Learning dynamical systems from partial observations, Second Workshop on Machine Learning and the Physical Sciences (NeurIPS 2019), Vancouver, Canada, February 2019.
Yunhao Ba, Guangyuan Zhao, and Achuta Kadambi, Blending diverse physical priors with neural networks, arXiv:1910.00201, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, Neural machine translation by jointly learning to align and translate, arXiv:1409.0473, 2016.
Wael Bahsoun, Ian Melbourne, and Marks Ruziboev, Variance continuity for Lorenz flows, Ann. Henri Poincaré 21 (2020), no. 6, 1873–1892. MR 4100921, DOI 10.1007/s00023-020-00913-5
Randall D. Beer, On the dynamics of small continuous-time recurrent neural networks, Adapt. Behav. 3 (1995), no. 4, 469–509, http://journals.sagepub.com/doi/10.1177/105971239500300405.
A. Bensoussan, J.-L. Lions, and G. Papanicolaou, Asymptotic analysis for periodic structures, AMS Chelsea Publishing, Providence, RI, 2011. Corrected reprint of the 1978 original [MR0503330]. MR 2839402, DOI 10.1090/chel/374
José Bento, Morteza Ibrahimi, and Andrea Montanari, Information theoretic limits on learning stochastic differential equations, 2011 IEEE International Symposium on Information Theory Proceedings, IEEE, 2011, pp. 855–859.
Tom Beucler, Michael Pritchard, Stephan Rasp, Jordan Ott, Pierre Baldi, and Pierre Gentine, Enforcing analytic constraints in neural networks emulating physical systems, Phys. Rev. Lett. 126 (2021), no. 9, Paper No. 098302, 7. MR 4231605, DOI 10.1103/physrevlett.126.098302
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart, Model reduction and neural networks for parametric PDEs, SMAI J. Comput. Math. 7 (2021), 121–157. MR 4290514, DOI 10.5802/smai-jcm.74
Marc Bocquet, Julien Brajard, Alberto Carrassi, and Laurent Bertino, Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization, Found. Data Sci. 2 (2020), no. 1, 55, https://www.aimsciences.org/article/doi/10.3934/fods.2020004.
Francesco Borra, Angelo Vulpiani, and Massimo Cencini, Effective models and predictability of chaotic multiscale systems via machine learning, Phys. Rev. E 102 (2020), no. 5, 052203, 11. MR 4189360, DOI 10.1103/physreve.102.052203
Julien Brajard, Alberto Carrassi, Marc Bocquet, and Laurent Bertino, Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model, J. Comput. Sci. 44 (2020), 101171, 11. MR 4117875, DOI 10.1016/j.jocs.2020.101171
Julien Brajard, Alberto Carrassi, Marc Bocquet, and Laurent Bertino, Combining data assimilation and machine learning to infer unresolved scale parametrization, Philos. Trans. Roy. Soc. A 379 (2021), no. 2194, Paper No. 20200086, 16. MR 4236154, DOI 10.1098/rsta.2020.0086
Leo Breiman, Bagging predictors, Mach. Learn., 24 (1996), no. 2, 123–140.
N. D. Brenowitz and C. S. Bretherton, Prognostic validation of a neural network unified physics parameterization, Geophys. Res. Lett. 45, no. 12, 6289–6298, 2018. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2018GL078510.
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci. USA 113 (2016), no. 15, 3932–3937. MR 3494081, DOI 10.1073/pnas.1517384113
Dmitry Burov, Dimitrios Giannakis, Krithika Manohar, and Andrew Stuart, Kernel analog forecasting: multiscale test problems, Multiscale Model. Simul. 19 (2021), no. 2, 1011–1040. MR 4272899, DOI 10.1137/20M1338289
Kathleen Champion, Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton, Data-driven discovery of coordinates and governing equations, Proc. Natl. Acad. Sci. USA 116 (2019), no. 45, 22445–22451. MR 4032517, DOI 10.1073/pnas.1906995116
Bo Chang, Minmin Chen, Eldad Haber, and Ed H. Chi, AntisymmetricRNN: A dynamical system view on recurrent neural networks, arXiv:1902.09689, 2019.
Ashesh Chattopadhyay, Pedram Hassanzadeh, and Devika Subramanian, Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network, Nonlinear Process. Geophys. 27 (2020), no. 3, 373–389, https://npg.copernicus.org/articles/27/373/2020/.
Ashesh Chattopadhyay, Adam Subel, and Pedram Hassanzadeh, Data-driven super-parameterization using deep learning: experimentation with multiscale Lorenz 96 systems and transfer learning. J. Adv. Model. Earth Sys. 12 (2020), no. 11, e2020MS002084, https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020MS002084.
Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M. Stuart, Solving and learning nonlinear PDEs with Gaussian processes, J. Comput. Phys. 447 (2021), Paper No. 110668, 29. MR 4311012, DOI 10.1016/j.jcp.2021.110668
Yuming Chen, Daniel Sanz-Alonso, and Rebecca Willett, Autodifferentiable ensemble Kalman filters, SIAM J. Math. Data Sci. 4 (2022), no. 2, 801–833. MR 4443668, DOI 10.1137/21M1434477
Oksana A. Chkrebtii, David A. Campbell, Ben Calderhead, and Mark A. Girolami, Bayesian solution uncertainty quantification for differential equations, Bayesian Anal. 11 (2016), no. 4, 1239–1267. MR 3577378, DOI 10.1214/16-BA1017
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio, On the properties of neural machine translation: encoder-decoder approaches, arXiv:1409.1259, 2014.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, Learning phrase representations using rnn encoder-decoder for statistical machine translation, arXiv:1406.1078, 2014.
Alexandre J. Chorin and Fei Lu, Discrete approach to stochastic parametrization and dimension reduction in nonlinear dynamics, Proc. Natl. Acad. Sci. 112 (2015), no. 32, 9804–9809, https://www.pnas.org/content/112/32/9804.
Alexandre J. Chorin, Ole H. Hald, and Raz Kupferman, Optimal prediction and the Mori-Zwanzig representation of irreversible processes, Proc. Natl. Acad. Sci. USA 97 (2000), no. 7, 2968–2973. MR 1750741, DOI 10.1073/pnas.97.7.2968
David Colton and Rainer Kress, Inverse acoustic and electromagnetic scattering theory, 3rd ed., Applied Mathematical Sciences, vol. 93, Springer, New York, 2013. MR 2986407, DOI 10.1007/978-1-4614-4942-3
Grace Wahba, Smoothing noisy data with spline functions, Numer. Math. 24 (1975), no. 5, 383–393. MR 405795, DOI 10.1007/BF01437407
G. Cybenko, Approximation by superpositions of a sigmoidal function, Math. Control Signals Systems 2 (1989), no. 4, 303–314. MR 1015670, DOI 10.1007/BF02551274
Eric Darve, Jose Solomon, and Amirali Kia, Computing generalized Langevin equations and generalized Fokker–Planck equations, Proc. Natl. Acad. Sci. 106 (2009), no. 27, 10884–10889.
Ronald A. DeVore and George G. Lorentz, Constructive approximation, Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 303, Springer-Verlag, Berlin, 1993. MR 1261635, DOI 10.1007/978-3-662-02888-9
Jonathan Dong, Ruben Ohana, Mushegh Rafayelyan, and Florent Krzakala, Reservoir computing meets recurrent kernels and structured transforms, arXiv:2006.07310, 2020.
J. R. Dormand and P. J. Prince, A family of embedded Runge-Kutta formulae, J. Comput. Appl. Math. 6 (1980), no. 1, 19–26. MR 568599, DOI 10.1016/0771-050X(80)90013-3
Qiang Du, Yiqi Gu, Haizhao Yang, and Chao Zhou, The discovery of dynamics via linear multistep methods and deep learning: error estimation, SIAM J. Numer. Anal. 60 (2022), no. 4, 2014–2045. MR 4464472, DOI 10.1137/21M140691X
Karthik Duraisamy, Gianluca Iaccarino, and Heng Xiao, Turbulence modeling in the age of data, Annual review of fluid mechanics. Vol. 51, Annu. Rev. Fluid Mech., vol. 51, Annual Reviews, Palo Alto, CA, 2019, pp. 357–377. MR 3965623
Weinan E, Chao Ma, and Lei Wu, A priori estimates of the population risk for two-layer neural networks, Commun. Math. Sci. 17 (2019), no. 5, 1407–1425. MR 4044196, DOI 10.4310/CMS.2019.v17.n5.a11
N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney, Lipschitz recurrent neural networks, arXiv:2006.12070, 2020.
Alban Farchi, Patrick Laloyaux, Massimo Bonavita, and Marc Bocquet, Using machine learning to correct model error in data assimilation and forecast applications, arXiv:2010.12605, 2021.
Ibrahim Fatkullin and Eric Vanden-Eijnden, A computational strategy for multiscale systems with applications to Lorenz 96 model, J. Comput. Phys. 200 (2004), no. 2, 605–638. MR 2095278, DOI 10.1016/j.jcp.2004.04.013
Brian A. Freno and Kevin T. Carlberg, Machine-learning error models for approximate solutions to parameterized systems of nonlinear equations, Comput. Methods Appl. Mech. Engrg. 348 (2019), 250–296. MR 3911069, DOI 10.1016/j.cma.2019.01.024
Roger Frigola, Yutian Chen, and Carl Edward Rasmussen, Variational Gaussian process state-space models, Adv. Neural Inform. Process. Systems 27 (2014), https://proceedings.neurips.cc/paper/2014/hash/139f0874f2ded2e41b0393c4ac5644f7-Abstract.html.
Ken-ichi Funahashi, Approximation theory, dynamical systems, and recurrent neural networks, Proceedings of the 5th International Colloquium on Differential Equations, Vol. 2 (Plovdiv, 1994) Sci. Cult. Technol. Publ., Singapore, 1995, pp. 51–58. MR 1656699
Daniel J. Gauthier, Erik Bollt, Aaron Griffith, and Wendson A. S. Barbosa, Next generation reservoir computing, Nat. Comm. 12 (2021), no. 1, 5564, ISSN 2041-1723.
Faheem Gilani, Dimitrios Giannakis, and John Harlim, Kernel-based prediction of non-Markovian time series, Phys. D 418 (2021), Paper No. 132829, 16. MR 4204505, DOI 10.1016/j.physd.2020.132829
R. González-García, R. Rico-Martínez, and I. G. Kevrekidis, Identification of distributed parameter systems: a neural net based approach, Comput. Chem. Eng. 22 (1998), S965–S968, https://linkinghub.elsevier.com/retrieve/pii/S0098135498001914.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep learning, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2016. MR 3617773
Georg A. Gottwald and Sebastian Reich, Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations, Chaos 31 (2021), no. 10, Paper No. 101103, 8. MR 4323018, DOI 10.1063/5.0066080
Georg A. Gottwald and Sebastian Reich, Supervised learning from noisy observations: combining machine-learning techniques with data assimilation, Phys. D 423 (2021), Paper No. 132911, 15. MR 4249157, DOI 10.1016/j.physd.2021.132911
Ayoub Gouasmi, Eric J. Parish, and Karthik Duraisamy, A priori estimation of memory effects in reduced-order models of nonlinear systems using the Mori-Zwanzig formalism, Proc. A. 473 (2017), no. 2205, 20170385, 24. MR 3710327, DOI 10.1098/rspa.2017.0385
Wojciech W. Grabowski, Coupling cloud processes with the large-scale dynamics using the cloud-resolving convection parameterization (CRCP), J. Atmos. Sci. 58 (2001), no. 9, 978–997, https://journals.ametsoc.org/view/journals/atsc/58/9/1520-0469_2001_058_0978_ccpwtl_2.0.co_2.xml.
Lyudmila Grigoryeva and Juan-Pablo Ortega, Echo state networks are universal, arXiv:1806.00797, 2018.
Geoffrey R. Grimmett and David R. Stirzaker, Probability and random processes, Oxford University Press, Oxford, 2020. Fourth edition [of 0667520]. MR 4229142
Abhinav Gupta and Pierre F. J. Lermusiaux, Neural closure models for dynamical systems, Proc. A. 477 (2021), no. 2252, Paper No. 20201004, 29. MR 4312509
Eldad Haber and Lars Ruthotto, Stable architectures for deep neural networks, Inverse Problems 34 (2018), no. 1, 014004, 22. MR 3742361, DOI 10.1088/1361-6420/aa9a90
Franz Hamilton, Alun L. Lloyd, and Kevin B. Flores, Hybrid modeling and prediction of dynamical systems, PLOS Comput. Biol. 13 (2017), no. 7, e1005655, https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005655.
Boumediene Hamzi and Houman Owhadi, Learning dynamical systems from data: a simple cross-validation perspective, part I: Parametric kernel flows, Phys. D 421 (2021), Paper No. 132817, 10. MR 4233447, DOI 10.1016/j.physd.2020.132817
John Harlim, Shixiao W. Jiang, Senwei Liang, and Haizhao Yang, Machine learning for prediction with missing dynamics, J. Comput. Phys. 428 (2021), Paper No. 109922, 22. MR 4199367, DOI 10.1016/j.jcp.2020.109922
Fabrício P. Härter and Haroldo Fraga de Campos Velho, Data assimilation procedure by recurrent neural network, Eng. Appl. Comput. Fluid Mech. 6 (2012), 224–233, https://doi.org/10.1080/19942060.2012.11015417.
Simon Haykin, José C. Príncipe, Terrence J. Sejnowski, and John McWhirter (eds.), New directions in statistical signal processing: from systems to brain, Neural Information Processing Series, MIT Press, Cambridge, MA, 2007. MR 2590576
Dan Hendrycks and Kevin Gimpel, Gaussian error linear units (gelus), https://arxiv.org/abs/1606.08415, 2016.
Carmen Hijón, Pep Español, Eric Vanden-Eijnden, and Rafael Delgado-Buscalioni, Mori–Zwanzig formalism as a practical computational tool, Faraday Discuss. 144 (2010), 301–322.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, Neural Comput. 9 (1997), 1735–1780.
Mark Holland and Ian Melbourne, Central limit theorems and invariance principles for Lorenz attractors, J. Lond. Math. Soc. (2) 76 (2007), no. 2, 345–364. MR 2363420, DOI 10.1112/jlms/jdm060
Herbert Jaeger, The “echo state” approach to analysing and training recurrent neural networks-with an erratum note’, German National Research Center for Information Technology GMD Technical Report, Bonn, Germany, January 2001, p. 148.
Xiaowei Jia, Jared Willard, Anuj Karpatne, Jordan S. Read, Jacob A. Zwart, Michael Steinbach, and Vipin Kumar, Physics-guided machine learning for scientific discovery: an application in simulating lake temperature profiles, ACM/IMS Trans. Data Sci. 2 (2021), no. 3, 20:1–20:26, https://doi.org/10.1145/3447814.
Shixiao W. Jiang and John Harlim, Modeling of missing dynamical systems: deriving parametric models using a nonparametric framework, Res. Math. Sci. 7 (2020), no. 3, Paper No. 16, 25. MR 4123391, DOI 10.1007/s40687-020-00217-4
Kadierdan Kaheman, Eurika Kaiser, Benjamin Strom, J. Nathan Kutz, and Steven L. Brunton, Learning discrepancy models from experimental data, arXiv:1909.08574, 2019.
Kadierdan Kaheman, Steven L. Brunton, and J. Nathan Kutz, Automatic differentiation to simultaneously identify nonlinear dynamics and extract noise probability distributions from data, Mach. Learn. Sci. Technol. 3 (2022), no. 1, 015031, https://doi.org/10.1088/2632-2153/ac567a.
Jari Kaipio and Erkki Somersalo, Statistical and computational inverse problems, Applied Mathematical Sciences, vol. 160, Springer-Verlag, New York, 2005. MR 2102218
J. Nagoor Kani and Ahmed H. Elsheikh, DR-RNN: a deep residual recurrent neural network for model reduction, arXiv:1709.00939, 2017.
K. Kashinath, M. Mustafa, A. Albert et al., Physics-informed machine learning: case studies for weather and climate modelling, Philos. Trans. Roy. Soc. A 379 (2021), no. 2194, Paper No. 20200093, 36. MR 4236152, DOI 10.1098/rsta.2020.0093
Rachael T. Keller and Qiang Du, Discovery of dynamics using linear multistep methods, SIAM J. Numer. Anal. 59 (2021), no. 1, 429–455. MR 4216874, DOI 10.1137/19M130981X
Felix P. Kemeth, Tom Bertalan, Nikolaos Evangelou, Tianqi Cui, Saurabh Malani, and Ioannis G. Kevrekidis, Initializing LSTM internal states via manifold learning, Chaos 31 (2021), no. 9, Paper No. 093111, 14. MR 4311431, DOI 10.1063/5.0055371
Marat F. Khairoutdinov and David A. Randall, A cloud resolving model as a cloud parameterization in the NCAR community climate system model: preliminary results, Geophys. Res. Lett. 28 (2001), no. 18, 3617–3620, https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2001GL013552.
Diederik P. Kingma and Jimmy Ba, Adam: A method for stochastic optimization, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, Conference Track Proceedings, arXiv:1412.6980, 2015.
Juš Kocijan, Modelling and control of dynamic systems using Gaussian process models, Advances in Industrial Control, Springer, Cham, 2016. MR 3410992, DOI 10.1007/978-3-319-21021-6
Milan Korda, Mihai Putinar, and Igor Mezić, Data-driven spectral analysis of the Koopman operator, Appl. Comput. Harmon. Anal. 48 (2020), no. 2, 599–629. MR 4047538, DOI 10.1016/j.acha.2018.08.002
K. Krischer, R. Rico-Martínez, I. G. Kevrekidis, H. H. Rotermund, G. Ertl, and J. L. Hudson, Model identification of a spatiotemporally varying catalytic reaction, AIChE J. 39 (1993), no. 1, 89–98, January 1993, http://doi.wiley.com/10.1002/aic.690390110.
S. Kullback and R. A. Leibler, On information and sufficiency, Ann. Math. Statistics 22 (1951), 79–86. MR 39968, DOI 10.1214/aoms/1177729694
Yury A. Kutoyants, Statistical inference for ergodic diffusion processes, Springer Series in Statistics, Springer-Verlag London, Ltd., London, 2004. MR 2144185, DOI 10.1007/978-1-4471-3866-2
I. E. Lagaris, A. Likas, D. I. Fotiadis, and C. V. Massalas, A hardware implementable non-linear method for the solution of ordinary, partial and integrodifferential equations, Mathematical methods in scattering theory and biomedical technology (Metsovo, 1997) Pitman Res. Notes Math. Ser., vol. 390, Longman, Harlow, 1998, pp. 110–126. MR 1795139
Kody Law, Abhishek Shukla, and Andrew Stuart, Analysis of the 3DVAR filter for the partially observed Lorenz’63 model, Discrete Contin. Dyn. Syst. 34 (2014), no. 3, 1061–1078. MR 3094561, DOI 10.3934/dcds.2014.34.1061
Kody Law, Andrew Stuart, and Konstantinos Zygalakis, Data assimilation, Texts in Applied Mathematics, vol. 62, Springer, Cham, 2015. A mathematical introduction. MR 3363508, DOI 10.1007/978-3-319-20325-6
Youming Lei, Jian Hu, and Jianpeng Ding, A hybrid model based on deep LSTM for predicting high-dimensional chaotic systems, arXiv:2002.00799, 2020.
Zhen Li, Hee Sun Lee, Eric Darve, and George Em Karniadakis, Computing the non-Markovian coarse-grained interactions derived from the Mori–Zwanzig formalism in molecular systems: application to polymer melts, J. Chem. Phys. 146, no. 1, 014104.
Zhong Li, Jiequn Han, Weinan E, and Qianxiao Li, On the curse of memory in recurrent neural networks: approximation and optimization analysis, arXiv:2009.07799, 2020.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar, Fourier neural operator for parametric partial differential equations, arXiv:2010.08895, 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar, Markov neural operators for learning chaotic systems, arXiv:2106.06898, 2021.
Kevin K. Lin and Fei Lu, Data-driven model reduction, Wiener projections, and the Koopman-Mori-Zwanzig formalism, J. Comput. Phys. 424 (2021), Paper No. 109864, 33. MR 4165611, DOI 10.1016/j.jcp.2020.109864
Ori Linial, Neta Ravid, Danny Eytan, and Uri Shalit, Generative ODE modeling with known unknowns, Proceedings of the Conference on Health, Inference, and Learning, CHIL ’21, New York, NY, USA, Association for Computing Machinery, April 2021, pp. 79–94, https://doi.org/10.1145/3450439.3451866.
E. Lorenz, Predictability—a problem partly solved, Proc. Seminar on Predictability, Reading, UK, ECMWF, 1996. https://ci.nii.ac.jp/naid/10015392260/en/.
Edward N. Lorenz, Deterministic nonperiodic flow, J. Atmospheric Sci. 20 (1963), no. 2, 130–141. MR 4021434, DOI 10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2
Robert J. Lovelett, José L. Avalos, and Ioannis G. Kevrekidis, Partial observations and conservation laws: gray-box modeling in biotechnology and optogenetics, Ind. Eng. Chem. Res. 59 (2020), no. 6, 2611–2620, https://doi.org/10.1021/acs.iecr.9b04507.
Fei Lu, Data-driven model reduction for stochastic Burgers equations, Entropy 22 (2020), no. 12, Paper No. 1360, 22. MR 4222910, DOI 10.3390/e22121360
Fei Lu, Kevin K. Lin, and Alexandre J. Chorin, Comparison of continuous and discrete-time data-based modeling for hypoelliptic systems, Commun. Appl. Math. Comput. Sci. 11 (2016), no. 2, 187–216. MR 3606402, DOI 10.2140/camcos.2016.11.187
Fei Lu, Kevin K. Lin, and Alexandre J. Chorin, Data-based stochastic model reduction for the Kuramoto-Sivashinsky equation, Phys. D 340 (2017), 46–57. MR 3582762, DOI 10.1016/j.physd.2016.09.007
Lu Lu, Pengzhan Jin, and George Em Karniadakis, DeepONet: learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators, arXiv:1910.03193, 2020.
Zhixin Lu, Brian R. Hunt, and Edward Ott, Attractor reconstruction by machine learning, Chaos 28 (2018), no. 6, 061104, 9. MR 3816936, DOI 10.1063/1.5039508
Mantas Lukoševičius and Herbert Jaeger, Reservoir computing approaches to recurrent neural network training, Comput. Sci. Rev. 3 (2009), no. 3, 127–149, August 2009, https://www.sciencedirect.com/science/article/pii/S1574013709000173.
Chao Ma, Jianchun Wang, and Weinan E, Model reduction with memory and the machine learning of dynamical systems, Commun. Comput. Phys. 25 (2019), no. 4, http://www.global-sci.com/intro/article_detail/cicp/12885.html.
Romit Maulik, Bethany Lusch, and Prasanna Balaprakash, Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders, Phys. Fluids 33 (2021), no. 3, 037106, March 2021, https://aip.scitation.org/doi/abs/10.1063/5.0039986.
Kevin McGoff, Sayan Mukherjee, Andrew Nobel, and Natesh Pillai, Consistency of maximum likelihood estimation for some dynamical systems, Ann. Statist. 43 (2015), no. 1, 1–29. MR 3285598, DOI 10.1214/14-AOS1259
Xiao-Li Meng and David van Dyk, The EM algorithm—an old folk-song sung to a fast new tune, J. Roy. Statist. Soc. Ser. B 59 (1997), no. 3, 511–567. With discussion and a reply by the authors. MR 1452025, DOI 10.1111/1467-9868.00082
Andrew C. Miller, Nicholas J. Foti, and Emily Fox, Learning insulin-glucose dynamics in the wild, arXiv:2008.02852, 2020.
Andrew C. Miller, Nicholas J. Foti, and Emily B. Fox, Breiman’s two cultures: you don’t have to choose sides, arXiv:2104.12219, 2021.
Jonas Mockus, Bayesian approach to global optimization, Mathematics and its Applications (Soviet Series), vol. 37, Kluwer Academic Publishers Group, Dordrecht, 1989. Theory and applications; With a 5.25-inch IBM PC floppy disk. MR 1111483, DOI 10.1007/978-94-009-0909-0
Kumpati S. Narendra and Kannan Parthasarathy, Neural networks and dynamical systems, Internat. J. Approx. Reason. 6 (1992), no. 2, 109–131, https://www.sciencedirect.com/science/article/pii/0888613X9290014Q.
Nicholas H. Nelsen and Andrew M. Stuart, The random feature model for input-output maps between Banach spaces, SIAM J. Sci. Comput. 43 (2021), no. 5, A3212–A3243. MR 4313849, DOI 10.1137/20M133957X
Duong Nguyen, Said Ouala, Lucas Drumetz, and Ronan Fablet, EM-like learning chaotic dynamics from noisy and partial observations, arXiv:1903.10335, 2019.
Murphy Yuezhen Niu, Lior Horesh, and Isaac Chuang, Recurrent neural networks in the eye of differential equations, arXiv:1904.12933, 2019.
Fernando Nogueira, Bayesian optimization: open source constrained global optimization tool for Python, 2014. https://github.com/fmfn/BayesianOptimization.
Paul A. O’Gorman and John G. Dwyer, Using machine learning to parameterize moist convection: potential for modeling of climate, climate change, and extreme events, J. Adv. Model. Earth Syst. 10 (2018), no. 10, 2548–2563, https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2018MS001351.
S. Ouala, D. Nguyen, L. Drumetz, B. Chapron, A. Pascual, F. Collard, L. Gaultier, and R. Fablet, Learning latent dynamics for partially observed chaotic systems, Chaos 30 (2020), no. 10, 103121, 17. MR 4164381, DOI 10.1063/5.0019309
Eric J. Parish and Karthik Duraisamy, A dynamic subgrid scale model for large eddy simulations based on the Mori-Zwanzig formalism, J. Comput. Phys. 349 (2017), 154–175. MR 3695240, DOI 10.1016/j.jcp.2017.07.053
Jaideep Pathak, Brian Hunt, Michelle Girvan, Zhixin Lu, and Edward Ott, Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach, Phys. Rev. Lett. 120 (2018), no. 2, 024102, https://link.aps.org/doi/10.1103/PhysRevLett.120.024102.
Jaideep Pathak, Alexander Wikner, Rebeckah Fussell, Sarthak Chandra, Brian R. Hunt, Michelle Girvan, and Edward Ott, Hybrid forecasting of chaotic processes: using machine learning in conjunction with a knowledge-based model, Chaos 28 (2018), no. 4, 041101, 9. MR 3787327, DOI 10.1063/1.5028373
Grigorios A. Pavliotis and Andrew M. Stuart, Multiscale methods, Texts in Applied Mathematics, vol. 53, Springer, New York, 2008. Averaging and homogenization. MR 2382139
Matthew Plumlee, Bayesian calibration of inexact computer models, J. Amer. Statist. Assoc. 112 (2017), no. 519, 1274–1285. MR 3735376, DOI 10.1080/01621459.2016.1211016
Matthew Plumlee and V. Roshan Joseph, Orthogonal Gaussian process models, Statist. Sinica 28 (2018), no. 2, 601–619. MR 3791080
Manuel Pulido, Pierre Tandeo, Marc Bocquet, Alberto Carrassi, and Magdalena Lucini, Stochastic parameterization identification using ensemble Kalman filtering combined with maximum likelihood methods, Tellus A Dyn. Meteorology Oceanogr. 70 (2018), no. 1, 1–17.
Ryan Pyle, Nikola Jovanovic, Devika Subramanian, Krishna V. Palem, and Ankit B. Patel, Domain-driven models yield better predictions at lower cost than reservoir computers in Lorenz systems, Philos. Trans. Roy. Soc. A 379 (2021), no. 2194, Paper No. 20200246, 22. MR 4236153, DOI 10.1103/physrevlett.120.024102
Zhaozhi Qian, William R. Zame, Lucas M. Fleuren, Paul Elbers, and Mihaela van der Schaar, Integrating expert ODEs into neural ODEs: pharmacology and disease progression, arXiv:2106.02875, 2021.
Alejandro F. Queiruga, N. Benjamin Erichson, Dane Taylor, and Michael W. Mahoney, Continuous-in-depth neural networks, arXiv:2008.02389, 2020.
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman, Universal differential equations for scientific machine learning, arXiv:2001.04385, 2020.
Christopher Rackauckas, Roshan Sharma, and Bernt Lie, Hybrid mechanistic + neural model of laboratory helicopter, pages 264–271, March 2021, pp. 264–271, https://ep.liu.se/en/conference-article.aspx?series=ecp&issue=176&Article_No=37.
Ali Rahimi and Benjamin Recht, Random features for large-scale kernel machines, Adv. Neural Inform. Process. Syst. 20 (2008), Curran Associates, Inc., https://proceedings.neurips.cc/paper/2007/file/013a006f03dbc5392effeb8f18fda755-Paper.pdf.
Ali Rahimi and Benjamin Recht, Uniform approximation of functions with random bases, 2008 46th Annual Allerton Conference on Communication, Control, and Computing, IEEE, 2008, pp. 555–561.
Ali Rahimi and Benjamin Recht, Weighted sums of random kitchen sinks: replacing minimization with randomization in learning, Nips, Citeseer, 2008, pp. 1313–1320.
M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys. 378 (2019), 686–707. MR 3881695, DOI 10.1016/j.jcp.2018.10.045
M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys. 378 (2019), 686–707. MR 3881695, DOI 10.1016/j.jcp.2018.10.045
Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian processes for machine learning, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2006. MR 2514435
Stephan Rasp, Michael S. Pritchard, and Pierre Gentine, Deep learning to represent subgrid processes in climate models, Proc. Natl. Acad. Sci. USA 115 (2018), no. 39, 9684–9689, https://www.pnas.org/content/115/39/9684.
Sebastian Reich and Colin Cotter, Probabilistic forecasting and Bayesian data assimilation, Cambridge University Press, New York, 2015. MR 3242790, DOI 10.1017/CBO9781107706804
R. Rico-Martines, I. G. Kevrekidis, M. C. Kube, and J. L. Hudson, Discrete- vs. continuous-time nonlinear signal processing: Attractors, transitions and parallel implementation issues, 1993 American Control Conference, June 1993, pp. 1475–1479, San Francisco, CA, USA, IEEE, ISBN 978-0-7803-0860-2, https://ieeexplore.ieee.org/document/4793116/.
R. Rico-Martínez, K. Krischer, I. G. Kevrekidis, M. C. Kube, and J. L. Hudson. Discrete- vs. continuous-time nonlinear signal processing of Cu electrodissolution data, Chemical Engineering Communications, 118 (1): 25–48, November 1992. https://www.tandfonline.com/doi/full/10.1080/00986449208936084.
R. Rico-Martinez, J. S. Anderson, and I. G. Kevrekidis, Continuous-time nonlinear signal processing: a neural network based approach for gray box identification, Proceedings of IEEE Workshop on Neural Networks for Signal Processing, Ermioni, Greece, IEEE, 1994, pp. 596–605. ISBN 978-0-7803-2026-0, http://ieeexplore.ieee.org/document/366006/.
Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud, Latent ODEs for irregularly-sampled time series, arXiv:1907.03907, 2019.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, Learning representations by back-propagating errors, Nature 323 (1986), no. 6088, 533–536.
Matteo Saveriano, Yuchao Yin, Pietro Falco, and Dongheui Lee, Data-efficient control policy search using residual dynamics learning, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), September 2017, pp. 4709–4715, ISSN 2153-0866
Hayden Schaeffer, Giang Tran, and Rachel Ward, Learning dynamical systems and bifurcation via group sparsity, Preprint, arXiv:1709.01558, 2017.
Hayden Schaeffer, Giang Tran, and Rachel Ward, Extracting sparse high-dimensional dynamics from limited data, SIAM J. Appl. Math. 78 (2018), no. 6, 3279–3295. MR 3885764, DOI 10.1137/18M116798X
Hayden Schaeffer, Giang Tran, Rachel Ward, and Linan Zhang, Extracting structured dynamical systems using sparse optimization with very few samples, Multiscale Model. Simul. 18 (2020), no. 4, 1435–1461. MR 4164069, DOI 10.1137/18M1194730
Anton Maximilian Schäfer and Hans-Georg Zimmermann, Recurrent neural networks are universal approximators, International Journal of Neural Systems 17, no. 4, 253–263, 2007.
Robert E. Schapire, The strength of weak learnability, Mach. Learn. 5 (1990), no. 2, 197–227.
Tapio Schneider, Andrew M. Stuart, and Jin-Long Wu, Learning stochastic closures using ensemble Kalman inversion, Trans. Math. Appl. 5 (2021), no. 1, Paper No. tnab003, 31. MR 4356471, DOI 10.1093/imatrm/tnab003
Skipper Seabold and Josef Perktold, statsmodels: econometric and statistical modeling with python, 9th Python in Science Conference, 2010.
Alex Sherstinsky, Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network, Phys. D 404 (2020), 132306, 28. MR 4057560, DOI 10.1016/j.physd.2019.132306
Guanya Shi, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung, Neural Lander: stable drone landing control using learned dynamics, arXiv:1811.08027, 2018.
Jonathan D. Smith, Kamyar Azizzadenesheli, and Zachary E. Ross, EikoNet: solving the eikonal equation with deep neural networks, IEEE Transactions on Geoscience and Remote Sensing, 2020, pp. 1–12.
Peter D. Sottile, David Albers, Peter E. DeWitt, Seth Russell, J. N. Stroh, David P. Kao, Bonnie Adrian, Matthew E. Levine, Ryan Mooney, Lenny Larchick, Jean S. Kutner, Matthew K. Wynia, Jeffrey J. Glasheen, and Tellen D. Bennett, Real-time electronic health record mortality prediction during the COVID-19 pandemic: A prospective cohort study, medRxiv, Cold Spring Harbor Laboratory Press, January 2021, p. 2021.01.14.21249793, https://www.medrxiv.org/content/10.1101/2021.01.14.21249793v1.
Langxuan Su and Sayan Mukherjee, A large deviation approach to posterior consistency in dynamical systems, arXiv:2106.06894, 2021.
Floris Takens, Detecting strange attractors in turbulence, Dynamical systems and turbulence, Warwick 1980 (Coventry, 1979/1980), Lecture Notes in Math., vol. 898, Springer, Berlin-New York, 1981, pp. 366–381. MR 654900
Zhihong Tan, Colleen M. Kaul, Kyle G. Pressel, Yair Cohen, Tapio Schneider, and João Teixeira, An extended eddy-diffusivity mass-flux scheme for unified representation of subgrid-scale turbulence and convection, J. Adv. Model. Earth Sys. 10 (2010), no. 3, 770–800.
Giang Tran and Rachel Ward, Exact recovery of chaotic systems from highly corrupted data, Multiscale Model. Simul. 15 (2017), no. 3, 1108–1129. MR 3678496, DOI 10.1137/16M1086637
Jonathan H. Tu, Clarence W. Rowley, Dirk M. Luchtenburg, Steven L. Brunton, and J. Nathan Kutz, On dynamic mode decomposition: theory and applications, J. Comput. Dyn. 1 (2014), no. 2, 391–421. MR 3415261, DOI 10.3934/jcd.2014.1.391
Eric Vanden-Eijnden, Numerical techniques for multi-scale dynamical systems with stochastic effects, Commun. Math. Sci. 1 (2003), no. 2, 385–391. MR 1980483, DOI 10.4310/CMS.2003.v1.n2.a11
Vladimir N. Vapnik, The nature of statistical learning theory, Springer-Verlag, New York, 1995. MR 1367965, DOI 10.1007/978-1-4757-2440-0
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, and Paul van Mulbregt, SciPy 1.0: fundamental algorithms for scientific computing in Python, Nat. Methods 17 (2020), no. 3, 261–272, https://www.nature.com/articles/s41592-019-0686-2.
P. R. Vlachas, J. Pathak, B. R. Hunt, T. P. Sapsis, M. Girvan, E. Ott, and P. Koumoutsakos, Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics, Neural Netw. 126 (2020), 191–217, https://linkinghub.elsevier.com/retrieve/pii/S0893608020300708.
Jack Wang, Aaron Hertzmann, and David J. Fleet, Gaussian process dynamical models, Adv. Neural Inform. Process. Syst. 18 (2005), https://papers.nips.cc/paper/2005/hash/ccd45007df44dd0f12098f486e7e8a0f-Abstract.html.
Qian Wang, Nicolò Ripamonti, and Jan S. Hesthaven, Recurrent neural network closure of parametric POD-Galerkin reduced-order models based on the Mori-Zwanzig formalism, J. Comput. Phys. 410 (2020), 109402, 32. MR 4079581, DOI 10.1016/j.jcp.2020.109402
Peter A. G. Watson, Applying machine learning to improve simulations of a chaotic dynamical system using empirical error correction, arXiv:1904.10904, 2019.
Alexander Wikner, Jaideep Pathak, Brian Hunt, Michelle Girvan, Troy Arcomano, Istvan Szunyogh, Andrew Pomerance, and Edward Ott, Combining machine learning with knowledge-based modeling for scalable forecasting and subgrid-scale closure of large, complex, spatiotemporal systems, Chaos 30 (2020), no. 5, 053111, 16. MR 4094693, DOI 10.1063/5.0005541
Jared Willard, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar, Integrating scientific knowledge with machine learning for engineering and environmental systems, arXiv:2003.04919, 2021.
J. A. Wilson and L. F. M. Zorzetto, A generalised approach to process state estimation using hybrid artificial neural network/mechanistic models, Comput. Chem. Eng. 21 (1997), no. 9, 951–963, http://linkinghub.elsevier.com/retrieve/pii/S0098135496003365.
Armand Wirgin, The inverse crime, arXiv:math-ph/0401050, 2004.
David H. Wolpert, Stacked generalization, Neural Netw. 5 (1992), no. 2, 241–259, https://www.sciencedirect.com/science/article/pii/S0893608005800231.
Zhong Yi Wan, Petr Karnakov, Petros Koumoutsakos, and Themistoklis P. Sapsis, Bubbles in turbulent flows: data-driven, kinematic models with history terms, Int. J. Multiph. Flow 129 (2020), 103286, 11. MR 4096711, DOI 10.1016/j.ijmultiphaseflow.2020.103286
Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari, Augmenting physical models with deep networks for complex dynamics forecasting, J. Stat. Mech. Theory Exp. 12 (2021), Paper No. 124012, 30. MR 4412840, DOI 10.1088/1742-5468/ac3ae5
He Zhang, John Harlim, Xiantao Li, Estimating linear response statistics using orthogonal polynomials: an RKHS formulation, Found. Data Sci. 2 (2020), no. 4, 443–485, http://aimsciences.org//article/doi/10.3934/fods.2020021.
He Zhang, John Harlim, and Xiantao Li, Error bounds of the invariant statistics in machine learning of ergodic Itô diffusions, Phys. D 427 (2021), Paper No. 133022, 28. MR 4311551, DOI 10.1016/j.physd.2021.133022
Jian Zhu and Masafumi Kamachi, An adaptive variational method for data assimilation with imperfect models, Tellus A Dyn. Meteorology Oceanogr. 52, (2000), no. 3, 265–279, https://doi.org/10.3402/tellusa.v52i3.12265.
Yuanran Zhu, Jason M. Dominy, and Daniele Venturi, On the estimation of the Mori-Zwanzig memory integral, J. Math. Phys. 59 (2018), no. 10, 103501, 39. MR 3856552, DOI 10.1063/1.5003467