Stability of minimal orbits

Abstract:

Let $G$ be a compact Lie group of isometries of a riemannian manifold $M$. It is well known that the minimal principal orbits are those on which the volume function ${\mathbf {v}}$, which assigns to $p \in M$ the volume of the orbit of $p$, is critical. It is shown that stability of a minimal orbit on which the hessian of ${\mathbf {v}}$ is nonnegative is determined by the degree of involutivity of the distribution of normal planes to the orbits. Specifically, if the lengths of the tangential components of Lie brackets of pairs of orthonormal normal vector fields are sufficiently small relative to the hessian of ${\mathbf {v}}$, then the minimal orbit is stable, and conversely. Computable lower bounds are obtained for the values of these parameters at which stability turns to instability. These lower bounds are positive even in the case where ${\mathbf {v}}$ is constant, and are finite unless the normal distribution is involutive. Several examples in which $M$ is a compact classical Lie group and $G$ is a subgroup of $M$ are discussed, showing in particular that the above estimates are sharp.