Direct integral decompositions and multiplicities for induced representations of nilpotent Lie groups

Abstract:

Let $K$ be a Lie subgroup of the connected, simply connected nilpotent Lie group $G$, and let $\mathfrak {k}$, $\mathfrak {g}$ be the corresponding Lie algebras. Suppose that $\sigma$ is an irreducible unitary representation of $K$. We give an explicit direct integral decomposition of ${\operatorname {Ind} _{k \to G}}\sigma$ into irreducibles. The description uses the Kirillov orbit picture, which gives a bijection between $G^\wedge$ and the coadjoint orbits in ${\mathfrak {g}^{\ast }}$ (and similarly for $K^\wedge , {\mathfrak {k}^{\ast }}$). Let $P:{\mathfrak {k}^{\ast }} \to {\mathfrak {g}^{\ast }}$ be the canonical projection, let ${\mathcal {O}_\sigma } \subset {\mathfrak {k}^{\ast }}$ be the orbit corresponding to $\sigma$, and, for $\pi \in G^\wedge$, let ${\mathcal {O}_\pi } \subset {\mathfrak {g}^{\ast }}$ be the corresponding orbit. The main result of the paper says essentially that $\pi \in G^\wedge$ appears in the direct integral iff ${P^{ - 1}}({\mathcal {O}_\sigma })$ meets ${\mathcal {O}_\pi }$; the multiplicity of $\pi$ is the number of ${\operatorname {Ad} ^{\ast }}(K)$-orbits in ${\mathcal {O}_\pi } \cap {P^{ - 1}}({\mathcal {O}_\sigma })$. There is also a natural description of the measure class in the integral.