Radon transform on symmetric matrix domains

Abstract:

Let $\mathbb {K}=\mathbb R, \mathbb C, \mathbb H$ be the field of real, complex or quaternionic numbers and $M_{p, q}(\mathbb {K})$ the vector space of all $p\times q$-matrices. Let $X$ be the matrix unit ball in $M_{n-r, r}(\mathbb {K})$ consisting of contractive matrices. As a symmetric space, $X=G/K=O(n-r, r)/O(n-r)\times O(r)$, $U(n-r, r)/U(n-r)\times U(r)$ and respectively $Sp(n-r, r)/Sp(n-r)\times Sp(r)$. The matrix unit ball $y_0$ in $M_{r^\prime -r, r}$ with $r^\prime \le n-1$ is a totally geodesic submanifold of $X$ and let $Y$ be the set of all $G$-translations of the submanifold $y_0$. The set $Y$ is then a manifold and an affine symmetric space. We consider the Radon transform $\mathcal Rf(y)$ for functions $f\in C_0^\infty (X)$ defined by integration of $f$ over the subset $y$, and the dual transform $\mathcal R^t F(x), x\in X$ for functions $F(y)$ on $Y$. For $2r <n, 2r\le r^\prime$ with a certain evenness condition in the case $\mathbb {K}=\mathbb {R}$, we find a $G$-invariant differential operator $\mathcal M$ and prove it is the right inverse of $\mathcal R^t \mathcal R$, $\mathcal R^t \mathcal R \mathcal M f=c f$, for $f\in C_0^\infty (X)$, $c\ne 0$. The operator $f\to \mathcal R^t\mathcal Rf$ is an integration of $f$ against a (singular) function determined by the root systems of $X$ and $y_0$. We study the analytic continuation of the powers of the function and we find a Bernstein-Sato type formula generalizing earlier work of the author in the set up of the Berezin transform. When $X$ is a rank one domain of hyperbolic balls in $\mathbb K^{n-1}$ and $y_0$ is the hyperbolic ball in $\mathbb K^{r^\prime -1}$, $1<r^\prime <n$ we obtain an inversion formula for the Radon transform, namely $\mathcal M\mathcal R^t\mathcal R f=c f$. This generalizes earlier results of Helgason for non-compact rank one symmetric spaces for the case $r^\prime =n-1$.