Asymptotic K-Support and Restrictions of Representations

In the late nineties T. Kobayashi wrote a series of papers in which he established a criterion for the discrete decomposablity of restrictions of unitary representations of reductive Lie groups to reductive subgroups. A key tool in the proof of sufficiency of his criterion was the use of the theory of hyperfunctions to study the microlocal behavior of characters of restrictions to compact subgroups. In this paper we show how to replace this tool by microlocal analysis in the $C^\infty$ category.


Introduction
In the late nineties T. Kobayashi wrote a series of papers in which he established a criterion for the discrete decomposablity of restrictions of unitary representations of reductive Lie groups to reductive subgroups.
A key tool in the proof of sufficiency of his criterion was the use of the theory of hyperfunctions to study the microlocal behavior of characters of restrictions to compact subgroups. See [6]. In this paper we show how to replace this tool by microlocal analysis in the C ∞ category.
In the following K denotes a connected, compact Lie group with Lie algebra k. We fix a maximal torus with Lie algebra t ⊂ k and an associated positive system. ByC ⊂ it * , i = √ −1, we denote the closure of the (dual) Weyl chamber. We identify equivalence classes of irreducible representations with their highest weights. Thus we writê K = Λ∩C, where Λ denotes the weight lattice in it * . We also assume an Ad-invariant inner product on k, extended to an Ad-invariant hermitian inner product on the complexification k C . We denote the norm of λ ∈ k C by |λ|. Using the inner products we identify t * and t * C with subsets of k * and of k * C , respectively. The Fourier series u = λ∈K u λ of any square integrable function u converges in L 2 (K). A (formal) Fourier series λ∈K u λ converges to a distribution u ∈ C −∞ (K) iff the L 2 norms u λ of the Fourier coefficients are polynomially bounded as functions of λ ∈K. Smooth functions, u ∈ C ∞ (K), are characterized by the rapid decrease of their Fourier coefficients, u λ = O(|λ| −∞ ) as λ → ∞. We shall define, for every distribution u, a closed cone afsupp(u) ⊂C \ 0, the asymptotic Fourier support of u. Essentially this is the smallest cone outside which the Fourier coefficients decrease rapidly. The asymptotic Fourier support is empty for C ∞ functions.
The wavefront set is a fundamental notion in the microlocal analysis of distributions. Given a closed cone Γ ⊂ T * K \ 0 one defines the space C −∞ Γ (K) which consists of all u ∈ C −∞ (K) having their wavefront sets contained in Γ, WF u ⊂ Γ. Under appropriate geometric conditions on Γ some operations can be extended by continuity to C −∞ Γ (K). The wavefront set was used by Howe [4] in a related setting.
The group K × K acts on the cotangent bundle T * K via left and right translations.
The Fourier series of u converges in C −∞ (K×K)·i −1 afsupp(u) (K). Kashiwara and Vergne [5, 4.5] proved the first assertion in the hyperfunction setting and noticed the C ∞ analogue in a remark. The importance of the second assertion is that it implies for subgroups satisfying geometric assumptions that restriction commutes with Fourier series.
A representation π of K in a Hilbert space is said to be polynomially bounded if the K-multiplicity m K (λ : π) = dim Hom K (λ, π) of λ in π is polynomially bounded as a function of λ ∈K. In particular, the multiplicities are finite then. The asymptotic K-support of π is a closed cone AS K (π) ⊂C \ 0 with approximates the support of m K (· : π) as λ → ∞. (See [6, (2.7.1)].) Theorem 2 ([6]). Let M be a closed subgroup of K. Denote its Lie algebra by m, and by m ⊥ ⊂ k * the space of conormals. Let π be a unitary representation of K which is polynomially bounded and which satisfies Then the restriction π| M of π to M is a polynomially bounded representation of M. The asymptotic M-support AS M (π| M ) is contained in the image of Ad * (K) AS K (π) under the canonical projection ik * → im * .
It is known that the restriction of an irreducible unitary representation π of a real reductive Lie group G to a maximal compact subgroup is polynomially bounded. For closed subgroups G ′ ⊂ G which are stable under the Cartan involution a criterion on G ′ -admissability of π| G ′ is given in [6,Theorem 2.9]. Theorem 2 contains all the micro-local information needed to rewrite the proof of [6, Theorem 2.9] without having to invoke the theory of hyperfunctions. Thus we offer an alternative approach to Kobayashi's theorem for readers without a strong background in hyperfunction theory.
The proof of Theorem 2 is centered around the notion of the Kcharacter λ∈K m K (λ : π) tr λ of π. The asumptions imply that this series converges in C −∞ (K), and that the K-character posesses a restriction to M which turns out to be the M-character of π| M . Theorem 1 is used to prove this. The continuity statement given in Theorem 1 simplifies the proof Theorem 2 when compared with the original argument.
The paper is organized as follows. In Section 2 we recall the expansion in eigenfunctions of a positive elliptic operator and its application to Fourier series on K. The asymptotic Fourier support is defined in this section. In Section 3 we study, for central distributions, wavefronts sets and the convergence of Fourier series. The theorems are proved in Sections 4 and 5.
This research grew out of the dissertation of the third author. The work was supported by the DFG via the international research training group "Geometry and Analysis of Symmetries".

Asymptotic Fourier support
The space C −∞ (K) of distributions on K is, by definition, the dual space of C ∞ (K). Functions are identified with distributions, L 2 (K) ⊂ C −∞ (K), using the normalized Haar measure dk on K. The L 2 scalar product (·|·) extends to an anti-duality between C −∞ (K) and C ∞ (K). We recall how the theory of Fourier series of distributions and of smooth functions follows from results on eigenfunction expansions of elliptic selfadjoint differential operators.
The Sobolev space H m (K) consists of all distributions which are mapped into L 2 (K) by differential operators with order ≤ m. We assume differential operators to be linear with C ∞ coefficients. H m (K) is equipped with a norm making it a Banach space. Let A be a second order, elliptic differential operator. Regard A as an unbounded operator on L 2 (K) with domain D(A) = H 2 (K). Its Hilbert space adjoint A * has, by elliptic regularity theory, the domain D(A * ) = H 2 (K). Assume, in addition, that (Au|u) > 0 if 0 = u ∈ D(A). Then A is positive selfadjoint. The eigenfunctions of A are in C ∞ (K).
Proposition 3 ([7, §10]). Let A be a positive selfadjoint second order elliptic differential operator on K. Let Au = j µ 2 j (u|e j )e j denote its spectral resolution where (e j ) ⊂ L 2 (K) is an orthonormal basis of eigenfunctions and 0 < µ j ↑ ∞ the corresponding sequence of eigenvalues of with the corresponding norm. The norm is equivalent with the graph norm. Hence D(A k ) is a Banach space. Obviously, H 2k (K) ⊂ D(A k ). By elliptic regularity we have equality, D(A k ) = H 2k (K). This holds also topologically because of Banach's theorem. By the Sobolev lemma, C ∞ (K) = ∩ k H 2k (K) as a projective limit. Hence the norms on D(A k ) define the Fréchet space topology of C ∞ (K). The asserted convergence criterion for C ∞ (K) follows from this. Using duality between weighted ℓ 2 sequence spaces we obtain the convergence criterion for C −∞ (K). Finally, the formula for the coefficients follows from the (separate) continuity of the antiduality bracket.
The ℓ 2 estimates in Proposition 3 can be replaced by supremum estimates because j µ −k j < ∞ for some k ∈ N. The latter property Denote by d λ , χ λ = tr λ, and M λ ⊂ L 2 (K) the dimension, the character and the space of matrix coefficients of λ ∈K. The convolution with d λ χ λ is the orthoprojector from L 2 (K) onto M λ . If u ∈ L 2 (K), then its Fourier series λ∈K u λ , u λ = d λ u * χ λ , converges to u in L 2 (K) by the Peter-Weyl theorem. The (formal) Fourier series of a distribution u ∈ C −∞ (K) is defined by the same formula using the convolution of a distribution with a C ∞ function, i.e., (u * ψ)(x) = K u(y)ψ(y −1 x) dy for ψ ∈ C ∞ (K) with the integral representing the duality bracket. Observe that χ λ , u * χ λ ∈ M λ ⊂ C ∞ (K). In general, we call a series λ∈K u λ with u λ ∈ M λ a Fourier series with coefficients u λ . We use left translation, L x (k) = xk, to trivialize the tangent bundle T K = K × k and the cotangent bundle T * K = K × k * . Under this identication left translation is the identity on the second components. Right translation R x (y) = yx acts, on the second components, as the adjoint action, dR x −1 : X → Ad(x)X, and as the co-adjoint action, Elements X ∈ U(k C ) of the universal enveloping algebra act as left invariant differential operators X on C −∞ (K). The principal symbol of the first order differential operator X associated with X ∈ k C is σ 1 ( X)(x, ξ) = ξ, X . Denote the Ad-invariant hermitian inner product on k C by Q. We assume that Q equals the negative Killing form on [k, k] and that the center of the Lie algebra is orthogonal to [k, k]. Choose, consistent with this orthogonal decomposition, an orthonormal basis Hence A is elliptic. It follows from the left invariance of X and the invariance of Haar measure that K Xv(y) dy = 0 for all v ∈ C ∞ (K), X ∈ k C . Therefore, A is positive selfadjoint with domain H 2 (K). Furthermore, A is bi-invariant. Therefore, each M λ , λ ∈K, is contained in an eigenspace of A with eigenvalue µ = µ(λ). There exists a constant C > 0 such that Here ρ is the half sum of positive roots. The left inequality holds because A − 1 is the sum of a non-negative operator B and the Casimir operator. It is well-known that the Casimir operator contains M λ in its eigenspace with eigenvalue |λ+ρ| 2 −|ρ| 2 . Since B is a sum of − X 2 , X ∈ t, the right inequality follows from (− X 2 u|u) = Xu 2 = λ, X u 2 which holds for any highest weight vector u ∈ M λ .
Summarizing we have the following.
for all, resp. for some, N ∈ Z. If u ∈ C −∞ (K), then its Fourier series Smoothness properties of a distribution correspond to decaying properties of its Fourier coefficients. We define an approximating cone to the directions of those λ ∈K ⊂C such that the Fourier coefficients u λ do not decay rapidly as λ → ∞. A subset of a (finite dimensional) real vector space V (or of a vector bundle) is called conic or a cone iff it is invariant under multiplication with positive reals.
Let u ∈ C −∞ (K) and λ∈K u λ its Fourier series. The asymptotic Fourier support of u is the closed cone afsupp(u) ⊂C \ 0 which is defined as follows. A point µ ∈C \ 0 is in the complement of afsupp(u) iff there is a conic neighbourhood S ⊂C \ 0 of µ such that λ∈S∩K |λ| 2N u λ 2 < ∞, ∀N ∈ N.
Remark 5. Instead of working with ℓ 2 -estimates we can work with supremum estimates such as sup λ∈S∩K |λ| N u λ < ∞. This follows from the observation made after the proof of Proposition 3.
With a subset S ⊂ V one associates the closed cone S ∞ ⊂ V \ 0 as follows. A point is in the complement of S ∞ if it has a conic neighbourhood which intersects S in a relatively compact set. Equivalently, v ∈ S ∞ iff there exist sequences (v j ) ⊂ S and ε j ↓ 0 such that lim j ε j v j = v. The cone S ∞ approximates S at infinity.
The K-support supp K (π) of a representation π of K in a Hilbert space is the set of all λ ∈K ⊂C such that λ occurs in π, i.e., m K (λ : π) > 0. The set AS K (π) = supp K (π) ∞ ⊂C \ 0 is the asymptotic K-support of π.

Wavefront convergence of central Fourier series
The definition of the wavefront set of a distribution is based on the calculus of pseudodifferential operators. We collect, in our context, some definitions and results, refering to [2, Section 2.5], [1], and [3, Section 18.1] for details.
With every pseudodifferential operator A ∈ Ψ m (K) one associates its set Char A ⊂ T * K \ 0 of characteristic points. A point is noncharacteristic if there is a symbol b ∈ S −m such that ab − 1 ∈ S −1 in a conic neighbourhood of that point. Here a ∈ S m (T * K) is, modulo S m−1 (T * K), a principal symbol of A. The operator is said to be elliptic at a non-characteristic point. An operator A : C ∞ (K) → C −∞ (K) is a pseudodifferential operator iff its Schwartz kernel K A ∈ C −∞ (K × K) is a conormal distribution respect to the diagonal. More explicitly, A ∈ Ψ m (K) iff the singular support of K A is contained in the diagonal and K can be covered with open sets U ⊂ K such that the kernel is given by an oscillatory integral (3) K A (y ′ , y) = e iϕ(y ′ ,η)−iϕ(y,η) a(y ′ , y, η) dη, y ′ , y ∈ U. The phase function ϕ ∈ C ∞ (U ×k) is real-valued, linear in the second variable, and nondegenerate, i.e., det ϕ ′′ yη = 0. The amplitude a belongs to the symbol space S m (U × U × k * ). A is elliptic at ξ = ϕ ′ x (x, ζ) ∈ T * x K \ 0, x ∈ U, iff there is a neighbourhood U 0 ⊂ U of x, a conic neighbourhood V of ζ, and C > 0 such that |a(y, y, η)| ≥ |η| m /C for y ∈ U 0 , η ∈ V , |η| > C.
Let u ∈ C −∞ (K). The wavefront set WF u ⊂ T * K \ 0 equals ∩ Char A, where the intersection is taken over all pseudodifferential operators A which satisfy Au ∈ C ∞ (K). Let Γ ⊂ T * K \ 0 be a closed cone. The space C −∞ Γ (K) of distributions on K which have their wavefront sets contained in Γ is equipped with a locally convex topology. It contains C ∞ (K) as a sequentially dense subspace. Convergence of a sequence, u j → u in C −∞ Γ (K), is equivalent to u j → u (weakly) in C −∞ (K) and the existence, for every (x, ξ) ∈ (T * K \ 0) \ Γ, of a pseudodifferential operator A ∈ Ψ m (K) such that (x, ξ) ∈ Char A, and Here WF(A) is the smallest conic subset of T * K \ 0 such that A is of order −∞ in the complement. (See the remark following Theorem 18.1.28 of [3].) Let K act on C ∞ (K) via the right regular representation, R x f (y) = f (yx). The corresponding action of the Lie algebra k C is by left invariant vector fields, dR e (X)f = Xf .
The following lemma should be compared with [5, 3.1].
Lemma 6. Let λ∈K u λ be a Fourier series which converges in C −∞ (K). Assume that each u λ is a highest weight vector for the right regular representation acting irreducibly on a subspace of M λ . Let S be a closed cone ⊂C \ 0. Then λ∈S∩K u λ converges in C −∞ K×i −1 S (K).
Proof. The differential equations Xu λ = 0 and Xu λ = λ, X u λ hold for X ∈ n and X ∈ t, respectively. Here n ⊂ k C denotes the sum of positive root spaces.
Let (x, ξ) ∈ K ×k * \0, ξ ∈ i −1 S. It suffices to find a pseudodifferential operator A, elliptic at (x, ξ), such that the series λ∈S∩K Au λ converges in C ∞ (K). If ξ ∈ t * , then there exists X ∈ n with ξ, X = 0; the first order differential operator A = X has the desired properties. Now assume ξ ∈ t * . Then the cone S − R + iξ is a closed subset of it * \ 0. It follows by a simple compactness argument that |λ| + |ξ| ≤ C|λ − iξ| with a constant C > 0 independent of λ ∈ S. Assume that S is convex. Choose X ∈ t which strictly separates the disjoint convex cones −R + ξ and iS. We infer that there exists c > 0 such that where Γ = R + ξ. By continuity (4) also holds in a conic neighbourhood Γ ⊂ k * \ 0 of ξ. Let U ⊂ K be an open neighbourhood of x and H ⊂ U a hypersurface containing x such that the following holds. The real vector field X is transversal to H and every maximally extended integral curve of X in U hits H in a unique point. Furthermore, y → exp −1 (x −1 y) maps U diffeomorphically onto an open neighbourhood of the origin in k. Using the method of characteristics we solve, for every η ∈ k * , the initial value problem The solution ϕ ∈ C ∞ (U × k * ) is linear in the second variable and ϕ ′ x (x, η) = η holds in T * x K = k * for all η. In particular, ϕ ′′ yη is nondegenerate at y = x. We have Xe −iϕ(·,η) u λ = λ − iη, X e −iϕ(·,η) u λ . Since X is left invariant, K Xv(y) dy = 0 holds for all v ∈ C ∞ (K). Therefore we can perform partial integration as follows, Iterating N times and estimating the integral on the right using the Cauchy-Schwarz inequality we obtain with a constant C N > 0 independent of λ ∈ S ∩K and η ∈ Γ. In view of (4) we get for all λ ∈ S ∩K and N ∈ N. Since the L 2 norms of the Fourier coefficients are polynomially bounded we obtain, for every χ ∈ C ∞ c (U), We can assume that, making U and Γ smaller if necessary, det ϕ ′′ yη = 0 in U × k * , and ϕ ′ y (U × Γ) ∩ i −1 S = ∅. Fix χ ∈ C ∞ c (U) with χ(x) = 1. Choose a symbol b ∈ S 0 (k * ) with supp b ⊂ Γ and b = 1 in a conic neighbourhood of ξ minus a compact set. Define the pseudodifferential operator A ∈ Ψ 0 (K) with kernel K A supported in U × U and given by (3) with amplitude a(y ′ , y, η) = χ(y ′ )b(η)χ(y). It follows from (5) that λ∈S∩K Au λ converges in C ∞ (K). Furthermore, A is elliptic at (x, ξ). Hence we have proved the assertion under the additional assumption that S is convex. To remove this assumption observe that S can be covered by finitely many closed convex cones each not containing iξ. Decompose the Fourier series correspondingly.
Pullback and pushforward of distributions is well-defined and continuous under assumptions on the wavefront sets. With any C ∞ map f : X → Y map between smooth manifolds one associates its canonical relation For a closed cone Γ ⊂ T * Y \ 0 define its pullback cone f * Γ = C −1 f • Γ ⊂ T * X. If f * Γ does not intersect the zero section, then the pullback f * u = u•f extends from C ∞ (X) to a (sequentially) continuous pullback operator f * : . If f is a proper map, then the pushforward operator f * : If, in addition, f is a submersion and Γ ⊂ T * X \ 0 is a closed cone, then f * Γ := C f • Γ ⊂ T * Y \ 0 and the pushforward restricts to a (sequentially) continuous map f * : . An important example of a pullback operator is the restriction to a submanifold M ⊂ K. It is defined on distributions having wavefront sets disjoint from the conormal bundle of M. The pushforward by a projection (x, y) → x is integration along fibers.
Taking the average Av f (x) = K f (yxy −1 ) dy of a function f extends uniquely from C ∞ (K) to an operator Av : . Proof. Define g : K × K → K, g(x, y) = yxy −1 , and p : K × K → K, p(x, y) = x. Then Av = p * g * on C ∞ (K). By assumption Γ = K × S where S ⊂ k * \ 0 is an Ad * -invariant closed cone. A computation shows that ((yxy −1 , ζ), (x, y, ξ, η)) ∈ C g iff ξ = Ad * (y −1 )ζ, and η = Ad * (xy −1 )ζ − Ad * (y −1 )ζ. Clearly, g * Γ does not intersect the zero section. Hence the pullback operator g * is defined. Composing C −1 g with the relation C p leads to η = 0 and p * g * Γ ⊂ K × Ad * (K)T . The assertion follows from this. Proof. For each λ ∈K we choose a highest weight vector w λ ∈ M λ of a irreducible subrepresentation ⊂ M λ of the right regular representation. We may view w λ as a matrix coefficient of the form w λ (x) = (R x v|v) with v ∈ M λ , v = 1. Then w λ (e) = 1, and w λ ≤ sup K |w λ | ≤ 1 by the Cauchy-Schwarz inequality. The central function Av w λ is a multiple of χ λ . Comparing values at e we get Av w λ = d −1 λ χ λ .
The dimension d λ and, in view of Corollary 4, the Fourier coefficients (u|χ λ ) of u grow at most polynomially in λ. Hence w = λ∈S∩K d λ (u|χ λ )w λ converges in C −∞ (K). By Lemma 6 the series converges in C −∞ K×i −1 S (K). The assertion follows from Lemma 7 since u = Av w.

Proof of Theorem 1
Every v ∈ C −∞ (K) defines a convolution operator C ∞ (K) → C ∞ (K), w → v * w. This is a continuous linear map which commutes with right translations. Conversely, every such map is given by convolution with a unique element v ∈ C −∞ (K). Composition of maps defines the convolution u The formula is evident for smooth functions and extends to distributions by separate sequential continuity. The composition C p • C −1 f of the canonical relations consists of all The wavefront of a tensor product satisfies Moreover, as a bilinear map the tensor product satisfies corresponding separate continuity properties. It follows that, for any two cones S 1 and S 2 in k * \ 0, the convolution (u 1 , u 2 ) → u 1 * u 2 defines a separately sequentially continuous bilinear map (6) * : . Convolution with the Dirac distribution δ = λ∈K d λ χ λ ∈ C −∞ (K) is the identity, δ * u = u. In the proof of the theorem we need δ S = λ∈S∩K d λ χ λ ∈ C −∞ (K) where S ⊂C \ 0. If S is a closed cone, then it follows from Proposition 8 that the series also converges to δ S in C −∞ K×i −1 Ad * (K)S (K). Now, turning to the proof of the theorem, let u ∈ C −∞ (K). Assume that S ⊂C \ 0 is a closed cone which contains afsupp(u) in its interior. Then the series of δC \S * u converges in C ∞ (K). Using (6) with S 1 = i −1 Ad * (K)S we deduce from the above that the Fourier series of δ S * u converges in C −∞ K×i −1 Ad * (K)S (K). It follows that the Fourier series of u = δ S * u + δC \S * u converges in this space, too. In particular, we have (K × K) · WF(u) ⊂ K × i −1 Ad * (K)S. This implies that the left-hand side in (1) is contained in the right-hand side.
To prove the opposite inclusion let S a closed cone ⊂C such that WF(u) ∩ (K × i −1 Ad * (K)S) = ∅. We apply (6) to δ S * u and deduce that the Fourier series λ∈S∩K u λ converges in C ∞ (K). This implies that S is disjoint from the asymptotic Fourier support of u. Since the closure of a Weyl chamber is a fundamental domain for the coadjoint action on k * , this implies K × i −1 afsupp(u) ⊂ (K × K) · WF u.

Proof of Theorem 2
The polynomial boundedness of π implies the finiteness of the multiplicities m K (λ : π) and the convergence of its K-character The support of Θ K π equals supp K (π). We have AS K (π) = supp K (π) ∞ = afsupp(Θ K π ). The second equality holds because the L 2 -norm of each non-zero summand in (7) is ≥ 1. ¿From Theorem 1 it follows that (7) converges in C −∞ Γ (K) where Γ = K × i −1 Ad * (K) AS K (π). Assumption (2) implies that the conormal bundle of M, which is a subset of K ×m ⊥ , is disjoint from Γ. Hence the restriction The assertions of the theorem will follow from this. Moreover, it says that Θ M π| M = Θ K π | M . Let µ ∈M . Fix a representation space H µ . Let ρ = ind K M (µ) denote the unitary representation of K induced by µ. We view the representation space H ρ of ρ as the subspace of L 2 (K, H µ ) defined by f (xm) = µ(m −1 )f (x), m ∈ M, almost every x ∈ K. Then f ∈ L 2 (K, H µ ) belongs to H ρ only if it satisfies, in the sense of distributions, the first order system of differential equations Y f + µ * (Y )f = 0, Y ∈ m. Here µ * is the Lie algebra representation induced by µ. The characteristic variety of Y + µ * (Y ) is contained in K × Y ⊥ . Hence WF(f ) ⊂ K × Ad * (K)m ⊥ if f ∈ H ρ . Theorem 1, generalized to vector valued distributions, implies that afsupp K (f ) ⊂ i Ad * (K)m ⊥ for every f ∈ H ρ . This implies AS K (ρ) ⊂ i Ad * (K)m ⊥ . Indeed, if this were not true, we could find a closed cone S ⊂C \ 0, S ∩ i Ad * (K)m ⊥ = ∅, and f = λ∈K∩S f λ ∈ H ρ , f λ ∈ W λ , such that λ∈K∩S |λ| 2N f λ 2 = ∞ for some N ∈ N. Here W λ denotes the λ-isotypical subspace of H ρ . Using assumption (2) we deduce AS K (ρ) ∩ AS K (π) = ∅. Therefore supp K (ρ) ∩ supp K (π) is relatively compact, hence finite. By Frobenius reciprocity we get, with sums having only finitely many nonzero summands,