On certain small representations of indefinite orthogonal groups

Abstract:

For any $n\in \mathbb {N}$ such that $2n\leq \min (p,q)$, we construct a representation $\pi _{n}$ of $O(p,q)$ with $p+q$ even as the kernel of a commuting set of $\frac {n(n+1)}{2}$ number of $O(p,q)$-invariant differential operators in the space of $C^{\infty }$ functions on an isotropic cone with a distinguished $GL_{n}(\mathbb {R})$-homogeneity degree. By identifying $\pi _{n}$ with a certain representation constructed via the formalism of the theta correspondence, we show (except when $p=q=2n$) that the space of $K$-finite vectors of $\pi _{n}$ is the $(\mathfrak {g},K)$-module of an irreducible unitary representation of $O(p,q)$ with Gelfand-Kirillov dimension $n(p+q-2n-1)$. Our construction generalizes the work of Binegar and Zierau (Unitarization of a singular representation of $SO_{e}(p,q)$, Commun. Math. Phys. 138 (1991), 245–258) for $n=1$.