Contractivity of neural ODEs: An eigenvalue optimization problem

Abstract:

We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ordinary differential equations (ODEs). When looking at contractivity properties of a one-layer weight-tied neural ODE $\dot {u}(t)=\sigma (Au(t)+b)$ (with $u,b \in \mathbb {R}^n$, $A$ is a given $n \times n$ matrix, $\sigma : \mathbb {R}\to \mathbb {R}$ denotes an activation function and for a vector $z \in \mathbb {R}^n$, $\sigma (z) \in \mathbb {R}^n$ has to be interpreted entry-wise), we are led to study the logarithmic norm of a set of products of type $D A$, where $D$ is a diagonal matrix such that $diag(D) \in \sigma ’(\mathbb {R}^n)$. Specifically, given a real number $c$ (usually $c=0$), the problem consists in finding the largest positive interval $\mathrm {I}\subseteq \mathbb [0,\infty )$ such that the logarithmic norm $\mu (DA) \le c$ for all diagonal matrices $D$ with $D_{ii}\in \mathrm {I}$. We propose a two-level nested methodology: an inner level where, for a given $\mathrm {I}$, we compute an optimizer $D^\star (\mathrm {I})$ by a gradient system approach, and an outer level where we tune $\mathrm {I}$ so that the value $c$ is reached by $\mu (D^\star (\mathrm {I})A)$. We extend the proposed two-level approach to the general multilayer, and possibly time-dependent, case $\dot {u}(t) = \sigma ( A_k(t) \ldots \sigma ( A_{1}(t) u(t) + b_{1}(t) ) \ldots + b_{k}(t) )$ and we propose several numerical examples to illustrate its behaviour, including its stabilizing performance on a one-layer neural ODE applied to the classification of the MNIST handwritten digits dataset.