A new family of irreducible subgroups of the orthogonal algebraic groups

Abstract

Let $n\geq 3,$ and let $Y$ be a simply connected, simple algebraic group of type $D_{n+1}$ over an algebraically closed field $K.$ Also let $X$ be the subgroup of type $B_n$ of $Y,$ embedded in the usual way. In this paper, we correct an error in a proof of a theorem of Seitz (Mem. Amer. Math. Soc. 67 (1987), no. 365), resulting in the discovery of a new family of triples $(X,Y,V),$ where $V$ denotes a finite-dimensional, irreducible, rational $KY$-module, on which $X$ acts irreducibly. We go on to investigate the impact of the existence of the new examples on the classification of the maximal closed connected subgroups of the classical algebraic groups.

1. Introduction

Let $K$ be an algebraically closed field of characteristic $p\geq 0.$ In the 1950s, Dynkin determined the maximal closed connected subgroups of the simple classical type linear algebraic groups defined over $K$, assuming $\operatorname {char}(K) = 0$ (see 1112); in 1987, Seitz 23 established an analogous classification in the case where $\operatorname {char}(K)>0$. The main step in both of these classifications is the determination of all triples $(X,Y,V),$ where $Y$ is a simple linear algebraic group defined over $K$, $X$ is a proper closed connected subgroup of $Y$, and $V$ is a non-trivial irreducible, finite-dimensional ($p$-restricted if $\operatorname {char}(K)=p>0$) $KY$-module on which $X$ acts irreducibly. The determination of these so-called “irreducible triples” is covered in the work of Dynkin 1112 (in case $\operatorname {char}(K)=0$), Seitz 23 (in case $\operatorname {char}(K) >0$ and $Y$ is a classical group), and Testerman 26 (in case $\operatorname {char}(K) >0$ and $Y$ is of exceptional type). The existence of an irreducible triple of the form $(X,Y,V)$ as above, arising from a rational representation $\rho : Y\rightarrow {\mathrm{GL}}(V)$, indicates that $\rho (X)$ is not maximal in the smallest classical group ${\mathrm{Isom}}(V)$ containing both $\rho (X)$ and $\rho (Y)$, while the large majority of tensor-indecomposable irreducible representations of a simple algebraic group give rise to maximal subgroups of the smallest classical group containing the image.

Recently, the second author’s PhD student Nathan Scheinmann discovered an irreducible triple which does not appear in 23, Theorem 1, Table 1. Namely, take $K$ to be of characteristic 3, and $X = B_3$ embedded in the usual way in $Y=D_4$ as the stabilizer of a non-singular 1-space on the 8-dimensional natural module for $Y$. Consider the irreducible $KY$-module with highest weight $\lambda _1+\lambda _2+\lambda _3$ (here $\lambda _i$, $1\leq i\leq 4$, is a set of fundamental weights for $D_4$, $\lambda _1$ is the highest weight of the natural $KD_4$-module, and we label Dynkin diagrams as in 3). The restriction of the highest weight to a maximal torus of $B_3$ shows the existence of a $B_3$-composition factor of highest weight $\omega _1+\omega _2+\omega _3$ ($\omega _i$, $i=1,2,3$, a set of fundamental weights for $B_3$) and consulting 19, one sees that these modules are both of dimension $384$, and hence $X$ acts irreducibly on $V$.

The absence of this example from 23, Table 1 is the result of an error in the proof of 23, 8.7. Here, we correct the error in this proof and, in so doing, establish the existence of a whole new family of modules $V$ for the group $Y=D_{n+1}$ on which $X=B_n$ acts irreducibly. For a fixed $n$ and a fixed $p$, there are finitely many modules $V$, but for each $n$ there exist infinitely many primes $p$ for which there is a new example. The precise description of the family is given in Theorem 1.2 below. In addition to this infinite family, our investigations revealed one further example of an irreducibly acting subgroup which does not appear in 23, Table 1, namely the group $Y=C_4$, defined over a field of characteristic $2$, acting on $V$, the irreducible module with highest weight $\lambda _3$, has a subgroup $X=B_3$, contained in a maximal rank subgroup of type $D_4\subset C_4$, both acting irreducibly on $V$. The triple $(D_4,C_4,V)$ appears in 23, Table 1, as well as the triple $(B_3,D_4, V|_{D_4})$. However, the triple $(B_3,C_4,V)$ is omitted.

The goal of this paper is two-fold: first we concentrate on the embedding $X=B_n\subset Y=D_{n+1}$ and determine all $p$-restricted irreducible representations of $Y$ whose restriction to $X$ is irreducible, thereby correcting 23, 8.7; see Theorem 1.2 below. The second goal of the paper is to show that the existence of the new examples has no further influence on the main results of 23 and 26. Indeed, the proofs of the main theorems in these two articles depend on an inductive hypothesis, concerning the list of examples for smaller rank groups. The new family of examples for the pair $(B_n,D_{n+1})$, as well as the one “new” example for the pair $(B_3,C_4),$ alters the inductive hypothesis and therefore requires one to take these new examples into consideration when working through all other possible embeddings. This is precisely what has been carried out in the proofs of Proposition 1.5, Proposition 1.7, and Theorem 1.8.

Remark 1.1.

Combining the results 23, Theorem 1, 26, Main Theorem, Theorem 1.2, Proposition 1.5, Proposition 1.7, and Theorem 1.8, we conclude that the only new examples to be added to 23, Table 1 are those for pairs $(X,Y)$ as follows:

•

$(B_n,D_{n+1}),$ $n\geq 3$, and irreducible modules described in Theorem 1.2, and

•

$(B_3,C_4)$, when $p=2$ and $B_3$ is embedded in a short root maximal rank $D_4$ subgroup of $C_4$, each acting irreducibly on the irreducible $KC_4$-module with highest weight $\lambda _3$. (Here $\lambda _i$, $1\leq i\leq 4$, is a set of fundamental weights for $C_4$ and we label Dynkin diagrams as in 3).

Theorem 1.2 covers the embedding $B_n\subset D_{n+1}$ and the case of $B_3\subset C_4$ is discussed at the end of Section 4. This assertion of the completeness of the rectified list is dependent upon our Hypothesis 1.4 below, where we state explicitly which results from 23 are assumed for the proofs of our main results.

Statement of results

Let $Y=\operatorname {Spin}_{2n+2}(K)$ $(n\geq 2)$ be a simply connected, simple algebraic group of type $D_{n+1}$ over $K.$ Also let $X$ be the subgroup of type $B_n,$ embedded in $Y$ in the usual way, as the stabilizer of a non-singular $1$-dimensional subspace of the natural module for $Y.$ Fix $T_Y$ a maximal torus of $Y$ and $B_Y\!\subset \! Y$ a Borel subgroup containing $T_Y$. Denote by $\{\lambda _1,\ldots ,\lambda _{n+1}\}$ the corresponding set of fundamental weights for $T_Y,$ ordered as in 3, where the natural $(2n+2)$-dimensional $KY$-module has highest weight $\lambda _1.$ Let $\sigma$ be a graph automorphism of $Y$ stabilizing $T_Y$, and with $X=Y^\sigma$, the group of $\sigma$-fixed points. Our first main result is the following; the proof is given in Section 3.

Theorem 1.2.

Let $Y=\operatorname {Spin}_{2n+2}(K)$ be a simply connected, simple algebraic group of type $D_{n+1}$ over $K$, $n\geq 2$, and let $X$ be the subgroup of type $B_n,$ as above. Consider a non-trivial, irreducible $KY$-module $V$ having $p$-restricted highest weight $\lambda =\sum _{r=1}^{n+1}{a_r\lambda _r}.$ Then $X$ acts irreducibly on $V$ if and only if $\lambda$ or $\sigma (\lambda )$ is equal to $\sum _{r=1}^n{a_r\lambda _r},$ with $a_n\neq 0,$ such that for all $1\leq i<j\leq n$ such that $a_ia_j\neq 0$ and $a_r=0$ for all $i<r<j,$ we have $p\mid (a_i+a_j +j-i).$

The set of weights which is listed in 23, Table 1 for the pair $(B_n,D_{n+1})$ is

$$\begin{equation*} \{c\lambda _n, a\lambda _k+b\lambda _n\ :\ abc\ne 0,\ p\mid (a+b+n-k)\}. \end{equation*}$$

So we see that the new examples are a generalization of those found by Seitz, where one congruence condition is replaced by a set of congruence conditions. (Note that there are new examples only if $n\geq 3$.) It is perhaps informative to point out precisely what error occurs in the proof of 23, 8.7, where the embedding $B_{n}\!\subset \! D_{n+1}$ is considered. In the proof, Seitz defines a certain vector in the irreducible $KY$-module of highest weight $\lambda$ and shows that this vector is annihilated by all simple root vectors in the Lie algebra of $X$, which then implies that $X$ does not act irreducibly. However if $\lambda$ satisfies the congruence conditions, the vector is in fact the zero vector in $V$ and so does not give rise to a second composition factor as claimed.

The omission of the triple $(B_3,C_4,\lambda _3)$ from 23, Table 1 is of a different nature, and occurs in the proof of 23, 15.13. In the first part of the proof, there is a reduction to the case where $X$ acts irreducibly on the natural $KY$-module with highest weight $\mu$, and the precise embedding we have here is for $\mu =\delta _3$ (following Seitz’s notation, $\delta _3$ is the third fundamental dominant weight), so $Y=C_4$ or $D_4$. Then Seitz argues that $\lambda =\lambda _k$ for some $k$. Since he is assuming that $V$ is not the natural $KY$-module, he invokes the inductive hypothesis and deduces that $\lambda =\lambda _3$. Now Seitz concludes that $V$ is a spin module for $Y$. Evidently, he considers only the embedding of $X$ in $D_4$, omitting to include the case of $X$ in $C_4$.

Returning to the family of examples described in Theorem 1.2, it is natural to ask how one might discover the given set of congruence conditions, and here we must give credit to the work of Ford in 13, where he studied irreducible triples of the form $(H,G,V)$, $G$ a simple classical type algebraic group over $K$, $H$ a disconnected closed subgroup of $G$ with $H^\circ$ simple, and $V$ an irreducible $KG$-module on which $H$ acts irreducibly. He discovered a family of irreducible triples for the embedding $D_n.2\subset B_n$, where the highest weight of the irreducible $KB_n$-module satisfies similar congruence conditions. His methods were later applied by Cavallin in 9 when studying irreducible $KB_n$-modules having precisely two $D_n$-composition factors.

The second goal of this article is to show that the existence of the new examples for the pair $(X,Y) = (B_n,D_{n+1})$, described by Theorem 1.2, and the further omitted example $(B_3,C_4)$ has no further influence on the main theorems in 2326. To explain the issue which must be addressed and our approach to the problem, we must describe to some extent the strategy of the proof of 23, Theorem 1. First note that the assumption that $X$ acts irreducibly on some $KY$-module implies that $X$ is semisimple. One of the main techniques used to determine the triples $(X,Y,V)$ as above involves arguing inductively, working with a suitable embedding $P_X=Q_XL_X\subset P_Y=Q_YL_Y$ of parabolic subgroups, where $Q_X = R_u(P_X)\subset R_u(P_Y) = Q_Y$. Indeed, 23, 2.1 implies that if $X$ acts irreducibly on $V,$ then the derived subgroup $L_X'$ acts irreducibly on the commutator quotient $V/[V,Q_Y]$, an irreducible $KL_Y'$-module. Moreover, the highest weight of $V/[V,Q_Y]$ as a $KL_Y'$-module is the restriction of the highest weight of $V$ to an appropriate maximal torus of $L_Y'$. (This is a variation of a result of 24.) Thus, Seitz and Testerman proceed by induction on the rank of $X$; Seitz treats the case $X$ of type $A_1$ by ad hoc methods, exploiting the fact that all weights of an irreducible $KA_1$-module are of multiplicity one. Now Theorem 1.2 above introduces a new family of examples of irreducible triples. As a consequence, one needs to reinvestigate all embeddings $X\subset Y$ where the pair $B_m\subset D_{m+1}$, $m\geq 3$, may arise when considering the projection of a Levi factor $L_X'$ of $X$ into a simple component of a Levi factor $L_Y'$ of $Y$, under the additional hypothesis that $X$ acts irreducibly on a $KY$-module whose highest weight has restriction to the $D_{m+1}$-component of $L_Y'$ among the new examples described by Theorem 1.2. This is precisely what we consider in Proposition 1.5 below. A similar analysis must be carried out for the one new example $(B_3,C_4)$ as a potential embedding of Levi factors. This easier case is covered by Proposition 1.7. In order to state the results, we introduce the following terminology.

Definition 1.3.

We will say a $p$-restricted dominant weight $\lambda$ for $Y=D_{m+1}$, $m\geq 3$, satisfies the congruence conditions if

(i)

$\lambda =\sum _{i=1}^{m}a_i\lambda _i$,

(ii)

$a_m\ne 0$,

(iii)

there exists $i< j\leq m-1$ with $a_ia_j\ne 0$, and

(iv)

for all $1\leq i<j\leq m$ such that $a_ia_j\ne 0$ and $a_r=0$ for all $i<r<j$, we have $p \mid (a_i+a_j+j-i)$.

Note that the above congruence conditions are precisely those satisfied by the highest weights in Theorem 1.2 but not appearing in 23, Table 1. (See the remark following the statement of Theorem 1.2.)

For the proofs of Proposition 1.5, Proposition 1.7, and Theorem 1.8, we require the following inductive hypotheses.

Hypothesis 1.4.

Assume $\operatorname {char}(K) = p>0$. Let $G$ be a simple algebraic group defined over $K$ and $H$ a semisimple, proper, closed subgroup of $G$, where the pair $(H,G)$ is one of the following:

(i)

$(H,B_n)$, $n\geq 3$,

(ii)

$(B_\ell ,A_n)$, $\ell \geq 2$,

(iii)

$(C_3,D_n)$, $n\geq 4$,

(iv)

$(C_3,A_5)$.

Let $V$ be a $p$-restricted irreducible $KG$-module, with corresponding representation $\rho :G\rightarrow \operatorname {GL}(V).$ Then $H$ acts irreducibly on $V$ if and only if the triple $(\rho (H),\rho (G),V)$ appears in 23, Table 1, where the highest weight of $V$ is given up to graph automorphisms of $G$.

The classical case

Let $Y$ be of classical type. The next two results ensure that, under the assumption of Hypothesis 1.4 for all embeddings $H\subset G$ with $\operatorname {rank}(G)<\operatorname {rank}(Y),$ the only new examples of irreducible triples $(X,Y,V)$ are those described in Remark 1.1.

Proposition 1.5.

Let $Y$ be a simply connected, simple algebraic group of type $D_{n+1}$, $n\geq 4$, and $X$ a semisimple, proper, closed subgroup of $Y$ acting irreducibly on a $p$-restricted irreducible $KY$-module $V$ of highest weight $\lambda$, $V$ not the natural module for $Y$. Assume Hypothesis 1.4 for all embeddings $H\subset G$ with $\operatorname {rank}(G)<\operatorname {rank}(Y).$ Moreover, if $X$ is simple, assume the following conditions are satisfied:

(i)

$P_X$ is a proper parabolic subgroup of $X$, with Levi factor $L_X$ of type $B_{m-1}$, $m\geq 4$.

(ii)

$P_Y$ is a parabolic subgroup of $Y$ with $P_X\subset P_Y$ and $R_u(P_X)\subset R_u(P_Y)$.

(iii)

For a Levi factor $L_Y$ of $P_Y$, writing $L_Y'=L_1L_2\cdots L_t$, a commuting product of simple groups, $L_t$ is of type $D_{m},$ and $L_X'$ projects non-trivially into $L_t.$

(iv)

For the irreducible $KL_Y'$-module $V/[V,Q_Y]$, write $V/[V,Q_Y]= M_1\otimes \cdots \otimes M_t$, where $M_i$ is an irreducible $KL_i$-module.

(v)

The highest weight of the $KL_t$-module $M_t$ satisfies the congruence conditions.

Then one of the following holds:

(a)

$X\subset B_k\times B_{n-k}$ for some $0<k<n,$ and $\lambda =\lambda _n$ or $\lambda _{n+1}$, or

(b)

$X=B_{n}$ and the embedding of $X$ in $Y$ is the usual embedding of $B_n$ in $D_{n+1}$, that is, $B_n$ is the stabilizer of a non-singular $1$-space on the natural $(2n+2)$-dimensional $KY$-module.

Moreover, in case (a) above, the subgroup $B_k\times B_{n-k}$ acts irreducibly on the $KY$-modules of highest weights $\lambda _n$ and $\lambda _{n+1}$.

Remark 1.6.

Now to go on to determine the irreducible triples satisfying (a), we rely on Hypothesis 1.4(i), and for those satisfying (b), we apply Theorem 1.2.

Proposition 1.7.

Assume $p={\mathrm{char}}(K) = 2$. Let $Y$ be a simply connected, simple algebraic group of type $C_n$, $n\geq 5$, and $X$ a semisimple, proper, closed subgroup of $Y$ with proper Levi factor of type $B_3$. Let $V$ be a $p$-restricted irreducible $KY$-module of highest weight $\lambda$, $V$ not the natural module for $Y$. Assume Hypothesis 1.4 for all embeddings $H\subset G$ with $\operatorname {rank}(G)<\operatorname {rank}(Y).$ Moreover, assume the following conditions are satisfied:

(i)

$P_X$ is a proper parabolic subgroup of $X$, with Levi factor $L_X$ of type $B_{3}$.

(ii)

$P_Y$ is a parabolic subgroup of $Y$ with $P_X\subset P_Y$ and $R_u(P_X)\subset R_u(P_Y)$.

(iii)

For a Levi factor $L_Y$ of $P_Y$, writing $L_Y'=L_1L_2\cdots L_t$, a commuting product of simple groups, $L_t$ is of type $C_4,$ and $L_X'$ projects non-trivially into $L_t.$

(iv)

For the irreducible $KL_Y'$-module $V/[V,Q_Y]$, write $V/[V,Q_Y]= M_1\otimes \cdots \otimes M_t$, where $M_i$ is an irreducible $KL_i$-module.

(v)

The highest weight of the $KL_t$-module $M_t$ is the third fundamental dominant weight for $L_t$.

Then $X$ acts reducibly on $V$.

The exceptional case

We now turn to the consideration of the case where $Y$ is a simply connected, simple algebraic group of exceptional type over $K$ and $X$ is a proper closed, connected subgroup of $Y$ acting irreducibly on some $p$-restricted irreducible $KY$-module. As usual, $X$ is then semisimple, and once again, we must consider the possibility of a parabolic embedding $P_X\subset P_Y,$ with Levi factor $L_X$ of $P_X,$ of type $B_m$, Levi factor $L_Y$ of $P_Y,$ having a simple factor of type $D_{m+1}$, with the action on the commutator quotient arising from a weight which satisfies the congruence conditions. (Note that the pair $(B_3,C_4)$ will never occur as an embedding of Levi factors when $Y$ is exceptional.) In particular, $Y$ is of type $E_n$ for $n=6,7$ or $8$.

Theorem 1.8.

Let $Y$ be a simply connected, simple algebraic group of type $E_n$, $6\leq n\leq 8$, defined over $K$ and let $X$ be a semisimple, proper, closed, connected subgroup of $Y,$ having a proper parabolic subgroup with Levi factor of type $B_m$ for some $m\geq 3$. Assume Hypothesis 1.4 for all embeddings $H\subset G$ with $\operatorname {rank}(G)<\operatorname {rank}(Y).$ Let $V$ be a non-trivial irreducible $KY$-module with $p$-restricted highest weight $\lambda$. Then $X$ acts irreducibly on $V$ if and only if $Y=E_6,$ $X=F_4,$ and one of the following holds:

(i)

$\lambda =(p-3)\lambda _1$ or $\lambda =(p-3)\lambda _6$, with $p>3.$

(ii)

$\lambda =\lambda _1+(p-2)\lambda _3$ or $\lambda =(p-2)\lambda _5+\lambda _6$, with $p>2.$

Note that the existence of the examples arising in Theorem 1.8 had already been established by Testerman 26, Main Theorem. The proof of the “only if” direction requires us to treat, eventually ruling out, several new potential configurations that arise from Theorem 1.2 in the inductive process, as explained in Section 5.

About the proofs

We conclude this section with a brief discussion of the methods and further remarks on our inductive assumption (Hypothesis 1.4). In order to prove Theorem 1.2, we first show that it is enough to work with the Lie algebras of $Y$ and $X.$ Indeed, as $\lambda$ is $p$-restricted, the irreducible $KY$-module $V$ is generated by a maximal vector $v^+$ for $B_Y$ as a module for the universal enveloping algebra $\mathfrak{U}_Y$ of $\operatorname {Lie}(Y).$ Therefore in order to show that $V|_X$ is irreducible, it suffices to show that $\mathfrak{U}_Yv^+ = \mathfrak{U}_Xv^+$, where $\mathfrak{U}_X$ is the universal enveloping algebra of $\operatorname {Lie}(X)$. We rely on the fact that any irreducible module for $Y$ is self-dual as a $KX$-module (see 3.1 below), and apply the techniques developed by Ford in 13, further investigated by Cavallin in 9, to establish this generation result.

For the proof of Proposition 1.5, we carry out an analysis used by Seitz in 23, Section 8, but applied specifically to the group $Y=D_{m+1}$. He first shows that a proper closed connected subgroup $X$ acts irreducibly on a non-trivial irreducible $KY$-module only if either $X$ acts irreducibly and tensor indecomposably on the natural module for $Y$, or the triple $(X,Y,V)$ is known. This part of our proof is not at all original, but we include it for completeness. At this point, however, our proof proceeds along different lines; we compare the commutator series for two different parabolic embeddings and obtain conditions on the highest weight which are compatible with the given congruence conditions only if the pair $(X,Y)$ is $(B_m,D_{m+1})$, which is handled by Theorem 1.2. The proof of Proposition 1.7 is much simpler given that we are dealing with a fixed-rank embedding.

For the proof of Theorem 1.8, we proceed differently than in 26; we use the classification of the maximal closed positive-dimensional subgroups of the exceptional type algebraic groups, given in 18, which was not available when 26 was written. Hence, we first consider the case where $X$ is maximal, find only the two examples of the theorem and conclude using the main result of 26 for the group $Y=F_4$.

In addition to Hypothesis 1.4, we rely upon two further results in 23, namely 23, Theorem 4.1 and 23, 6.1. The first result classifies the irreducible triples $(X,Y,V)$ when ${\mathrm{rank}}(X)={\mathrm{rank}}(Y)$, the second covers the case where ${\mathrm{rank}}(X)=1$. The proofs of these results are completely independent of the results in 23, Section 8. Finally, we will use the results of 23, Section 2 concerning parabolic embeddings and commutator series in irreducible modules for semisimple groups.

2. Preliminaries

In this section, we introduce the notation that shall be used in the remainder of the paper, and recall some basic properties of rational modules for simple linear algebraic groups. We rely on the standard reference 16 for a treatment of this general theory.

2.1. Notation

Let $K$ be an algebraically closed field of characteristic $p \geq 0,$ and let $G$ be a simply connected, simple linear algebraic group over $K.$ (All algebraic groups considered here will be linear algebraic groups, even if we omit to say so explicitly.) Also fix a Borel subgroup $B=UT$ of $G,$ where $T$ is a maximal torus of $G$ and $U$ denotes the unipotent radical of $B.$ Let $\operatorname {rank}(G)=\ell$ and let $\Pi =\{\alpha _1,\ldots , \alpha _\ell \}$ be the corresponding base of the root system $\Phi =\Phi ^+\sqcup \Phi ^-$ of $G,$ where $\Phi ^+$ and $\Phi ^-$ denote the sets of positive and negative roots, respectively. Throughout we use the ordering of simple roots as in 3. Let $\mathscr{W}$ be the Weyl group of $G,$ $\mathscr{W} = N_G(T)/T$, and for $\alpha \in \Phi ,$ denote by $s_{\alpha }$ the corresponding reflection. In addition, let

$$\begin{equation*} X(T)=\operatorname {Hom}(T,K^*) \end{equation*}$$

denote the character group of $T$ and write $(-,-)$ for a fixed $\mathscr{W}$-invariant inner product on the space $X(T)_{\mathbb{R}}=X(T)\otimes _{\mathbb{Z}} \mathbb{R}.$ Also let $\lambda _1,\ldots ,\lambda _\ell$ be the fundamental dominant weights for $T$ corresponding to our choice of base $\Pi ,$ that is, $\langle \lambda _i,\alpha _j\rangle =\delta _{ij}$ for $1\leq i\leq j\leq \ell ,$ where

$$\begin{equation*} \langle \lambda , \alpha \rangle =\frac{2(\lambda ,\alpha )}{(\alpha ,\alpha )} \end{equation*}$$

for $\lambda ,\alpha \in X(T)$, $\alpha \ne 0$. Set $X^+(T)=\{\lambda \in X(T):\langle \lambda ,\alpha \rangle \geq 0 \text{ for all } \alpha \in \Pi \}$ and call a character $\lambda \in X^+(T)$ a dominant $T$-weight (or simply dominant weight, if the choice of torus is clear in the context). Finally, we say that $\mu \in X(T)$ is under $\lambda \in X(T)$ (and we write $\mu \preccurlyeq \lambda$) if $\lambda -\mu =\sum _{r=1}^\ell {c_r\alpha _r}$ for some $c_1,\ldots ,c_\ell \in \mathbb{Z}_{\geq 0}.$ We also write $\mu \prec \lambda$ to indicate that $\mu \preccurlyeq \lambda$ and $\mu \neq \lambda .$

2.2. Rational modules

In this section, we recall some elementary facts on weights and multiplicities, as well as basic properties of Weyl and irreducible modules for $G.$ Let $V$ be a finite-dimensional, rational $KG$-module. Then

$$\begin{equation*} V=\bigoplus _{\mu \in X(T)}V_\mu , \end{equation*}$$

where, for $\mu \in X(T),$ $V_\mu =\{v\in V:tv=\mu (t)v \text{ for all } t\in T\}.$ A weight $\mu \in X(T)$ is called a weight of $V$ if $V_\mu \neq 0,$ in which case $V_\mu$ is said to be its corresponding weight space. Also, we denote by $\operatorname {m}_V(\mu )$ the multiplicity of $\mu$ in $V,$ and let $\Lambda (V)=\{\mu \in X(T):V_\mu \neq 0\}$ denote the set of weights of $V$ and write $\Lambda ^+(V)=\Lambda (V)\cap X^+(T)$ for the set of dominant weights of $V.$ It is well known that each weight of $V$ is $\mathscr{W}$-conjugate to a unique dominant weight in $\Lambda ^+(V).$ Also, if $\lambda \in X^+(T),$ then $w\lambda \preccurlyeq \lambda$ for every $w\in \mathscr{W}$, and all weights in a $\mathscr{W}$-orbit have the same multiplicity.

A non-zero vector $v^+\in V$ is called a maximal vector of weight $\lambda \in \Lambda (V)$ for the pair $(B,T)$ if $v^+\in V_\lambda$ and $Bv^+\subseteq \langle v^+\rangle _K.$ Now for $\lambda \in X^+(T)$ a dominant weight, we write $V_G(\lambda )$ for the Weyl module having highest weight $\lambda ,$ and denote by $L_G(\lambda )$ the unique irreducible quotient of $V_G(\lambda ).$ In other words,

$$\begin{equation*} L_G(\lambda )= V_G(\lambda )/\operatorname {rad}(\lambda ), \end{equation*}$$

where $\operatorname {rad}(\lambda )$ is the unique maximal submodule of $V_G(\lambda ),$ called the radical of $V_G(\lambda ).$ We write $\Lambda (\lambda )$ for $\Lambda (V_G(\lambda ))$ and $\Lambda ^+(\lambda )$ for $\Lambda ^+(V_G(\lambda )).$ Also, we denote by $H^0(\lambda )$ the induced $KG$-module having highest weight $\lambda .$ Recall that $H^0(\lambda )$ has a unique simple submodule, isomorphic to $L_G(\lambda ),$ and that

$$\begin{equation*} \Lambda (H^0(\lambda ))=\Lambda (-w_0\lambda ), \end{equation*}$$

where $w_0$ denotes the longest element in $\mathscr{W}.$ For $\mu \in X^+(T),$ we write $[V,L_G(\mu )]$ to denote the number of times the irreducible $KG$-module $L_G(\mu )$ appears as a composition factor of $V.$ We also use the notation

$$\begin{equation*} U_{\alpha }=\{x_\alpha (c):c\in K\} \end{equation*}$$

to denote the $T$-root subgroup of $G$ corresponding to the root $\alpha \in \Phi$ (that is, $x_\alpha :K\rightarrow G$ is a morphism of algebraic groups inducing an isomorphism onto $\operatorname {Im}(x_\alpha )$, such that $tx_\alpha (c) t^{-1}=x_\alpha (\alpha (t)c)$ for $t\in T$ and $c\in K$). Finally, we fix a Chevalley basis $\mathscr{B}=\{f_{\alpha },h_r,e_{\alpha } : \alpha \in \Phi ^+, 1\leq r\leq \ell \}$ for the Lie algebra $\operatorname {Lie}(G)$ of $G,$ compatible with our choice of $T\subset B,$ where $e_{\alpha }\in \operatorname {Lie}(G)_{\alpha },$ $f_{\alpha }\in \operatorname {Lie}(G)_{-\alpha }$ are root vectors for $\alpha \in \Phi ^+$ and $h_r=[e_{\alpha _r},f_{\alpha _r}]$ for $1\leq r\leq \ell .$ The proof of the following result can be deduced from applying the Poincaré-Birkhoff-Witt Theorem 4 to 10, A. 6.4.

Lemma 2.1.

Let $\lambda \in X^+(T)$ be a $p$-restricted dominant weight for $T,$ and let $V=L_G(\lambda )$. Also let $v^+\in V$ be a maximal vector of weight $\lambda$ for $B,$ and let $\mu \in \Lambda (V)$. Then for any fixed ordering $\leq$ on $\Phi ^+,$ we have

$$\begin{equation*} V_\mu \!=\!\left\langle f_{\gamma _1}\cdots f_{\gamma _k}v^+ : k\!\in \! \mathbb{Z}_{\geq 0},\nobreakspace \gamma _1,\cdots ,\gamma _k\!\in \! \Phi ^+,\nobreakspace \gamma _1\leq \cdots \leq \gamma _k,\nobreakspace \sum _{r=1}^k{\gamma _r}=\lambda -\mu \right\rangle _K. \end{equation*}$$

We conclude this section by illustrating how Lemma 2.1 can provide information on weight multiplicities in certain irreducible $KG$-modules in the case where $G$ is of type $A_\ell$ $(\ell \geq 2)$ over $K.$ Consider the dominant $T$-weights $\lambda =a\lambda _1+b\lambda _\ell$ $(a,b\geq 1)$ and $\mu =(a-1)\lambda _1+(b-1)\lambda _\ell .$ Writing $V=L_G(\lambda ),$ an application of Lemma 2.1 then shows that $V_\mu$ is spanned by

$$\begin{equation} \{f_{\alpha _1+\cdots +\alpha _r}f_{\alpha _{r+1}+\cdots + \alpha _\ell }v^+,1\leq r\leq \ell -1\}\cup \{f_{\alpha _1+\cdots +\alpha _\ell }v^+\}, {\tag{1}} \end{equation}$$

where $v^+$ is a maximal vector in $V$ for $B.$ (We used the fact that $f_{\alpha _i+\dots +\alpha _j}v^+=0$ for $1< i\leq j < \ell$ together with the commutator formula.) Finally, we set

$$\begin{equation*} V_{1,\ell } = \langle f_{\alpha _1+\cdots +\alpha _r}f_{\alpha _{r+1}+\cdots +\alpha _\ell } v^+ : 1\leq r\leq \ell -1\rangle _K. \end{equation*}$$

Proposition 2.2.

Assume $G$ is of type $A_\ell$ over $K$ for some $\ell \in \mathbb{Z}_{\geq 2}$ and consider the dominant $T$-weight $\lambda =a\lambda _1+b\lambda _\ell \in X^+(T),$ $1\leq a,b<p.$ Set $V = L_G(\lambda )$. Also let $\mu =(a-1)\lambda _1+(b-1)\lambda _\ell .$ Then the following assertions are equivalent:

(i)

The weight $\mu$ affords the highest weight of a composition factor of $V_G(\lambda ).$

(ii)

The inequality $\operatorname {m}_{V_G(\lambda )}(\mu ) > \operatorname {m}_V(\mu )$ is satisfied.

(iii)

The generators in 1 are linearly dependent.

(iv)

The element $f_{\alpha _1+\cdots + \alpha _\ell }v^+$ belongs to $V_{1,\ell }.$

(v)

The divisibility condition $p\mid a+b+\ell -1$ is satisfied.

Proof.

Clearly i implies ii. Conversely, the only dominant weights $\nu \in \Lambda ^+(V)$ such that $\mu \prec \nu \prec \lambda$ are $\lambda -\alpha _1$ (if $a>1$) and $\lambda -\alpha _\ell$ (if $b>1$). These weights have multiplicity $1$ in $V_G(\lambda )$ and hence none of them can afford the highest weight of a composition factor of $V_G(\lambda )$ by 22. Consequently ii implies i. Now an application of Freudenthal’s formula yields $\operatorname {m}_{V_G(\lambda )}(\mu )=\ell ,$ thus showing that $\ell = \operatorname {m}_{V_G(\lambda )}(\mu ) > \operatorname {m}_V(\mu )$ if and only if the $\ell$ generators in 1 are linearly dependent. Therefore ii and iii are equivalent as well. Finally, let $(\eta _r)_{1\leq r\leq \ell } \in K^\ell$ and set

$$\begin{equation*} w^+= \sum _{r=1}^{\ell -1}{\eta _r f_{\alpha _1+\cdots +\alpha _r}f_{\alpha _{r+1}+\cdots +\alpha _\ell }v^+} + \eta _\ell f_{\alpha _1+\cdots +\alpha _\ell }v^+. \end{equation*}$$

A straightforward calculation shows that $Bw^+=\langle w^+\rangle _K$ if and only if $p \mid a + b + \ell -1$ and $\eta _\ell \neq 0,$ in which case $w^+=0$ (since $V$ is irreducible and $w^+ \notin V_\lambda ).$ This shows that iii, iv, and v are equivalent, thus completing the proof.

■

3. Proof of Theorem 1.2

Let $K$ be an algebraically closed field having characteristic $p\geq 0$ and let $Y=\operatorname {Spin}_{2n+2}(K)$ be a simply connected, simple algebraic group of type $D_{n+1}$ over $K,$ with $n\geq 2.$ Let $X\subset Y$ be the subgroup of type $B_n,$ embedded in the usual way, as the stabilizer of a non-singular $1$-dimensional subspace of the natural $(2n+2)$-dimensional module for $Y.$ Fix $T_Y$ a maximal torus of $Y$ and $T_X$ a maximal torus of $X$ such that $T_X\subset T_Y$ and let $T_Y \subset B_Y,$ $T_X \subset B_X$ denote Borel subgroups of $Y,X$, respectively, with $B_X\subset B_Y.$ Let $\Pi (Y)=\{\alpha _1,\ldots ,\alpha _{n+1}\}$ be the corresponding base for the root system of $Y,$ and denote by $\{\lambda _1,\ldots ,\lambda _{n+1}\}$ the corresponding set of fundamental weights for $T_Y,$ where the natural $KY$-module has highest weight $\lambda _1.$ Let $\sigma$ be a graph automorphism of $Y$ stabilizing $T_Y,$ and with $X=Y^\sigma$, the group of $\sigma$-fixed points. Finally, let $\Pi (X)=\{\beta _1,\ldots ,\beta _n\}$ be the base for the corresponding root system of $X,$ associated with the choice of Borel subgroup $B_X$, and denote by $\{\omega _1,\ldots ,\omega _n\}$ the associated set of fundamental dominant weights for $T_X.$

3.1. Preliminary considerations

For $\sigma$ as above and for a $KY$-module $V,$ let $^\sigma V$ denote the vector space $V$ equipped with the $Y$-action $gv=\sigma (g)v$ for $g\in Y,$ $v\in V.$ Clearly $^\sigma V$ is irreducible if and only if $V$ is.

Lemma 3.1.

Let $V$ be an irreducible, finite-dimensional, rational $KY$-module. Then $V|_X$ is self-dual.

Proof.

Let $\lambda \in X^+(T_Y)$ be the highest weight of $V.$ Then $V^*$ has highest weight $-w_0 \lambda .$ If $n+1$ is even, then $-w_0=1$ by 25, Exercise 78 and so $V$ is self-dual as a $KY$-module, from which the desired result follows. If on the other hand $n+1$ is odd, then $^\sigma V \cong V^*$ by 25, Lemma 78, yielding $(V |_X)^* =(V^*)|_X \cong (^\sigma V)|_X = V|_X,$ since $\sigma (x) =x$ for all $\in X$. The result follows.

■

Let $\mathscr{B}_Y=\{f_{\alpha },h_r,e_{\alpha } : \alpha \in \Phi ^+(Y), 1\leq r\leq n+1\}$ be a Chevalley basis for the Lie algebra $\operatorname {Lie}(Y)$ of $Y,$ compatible with our choice of $T_Y\subset B_Y,$ as in Section 2.2. As in 23, Section 8, we may assume that the $B_n$-type subalgebra $\operatorname {Lie}(X)$ of $\operatorname {Lie}(Y)$ is generated by the root vectors

$$\begin{align} e_{\beta _r} &= e_{\alpha _r} \text{ for $1\leq r\leq n-1,$ }\\ e_{\beta _n} &= e_{\alpha _n} +e_{\alpha _{n+1}}, {\tag{2}}\\ f_{\beta _r} &= f_{\alpha _r} \text{ for $1\leq r\leq n-1,$ }\\ f_{\beta _n} &= f_{\alpha _n}+f_{\alpha _{n+1}}. \end{align}$$

In particular, we get that $\alpha _j|_{T_X}=\beta _j$ for $1\leq j \leq n-1,$ while $\alpha _{n+1}|_{T_X}=\alpha _n|_{T_X}=\beta _n,$ so that $\lambda _j|_{T_X}=\omega _j$ for $1\leq j\leq n-1,$ $\lambda _n|_{T_X}=\lambda _{n+1}|_{T_X}=\omega _n$ by 14, Section 13.2, p. 69. Also for $1\leq i\leq j \leq n+1,$ write $f_{i,j}=f_{\alpha _i+\cdots +\alpha _j},$ where we set $f_{i,i}=f_{\alpha _i}$ for $1\leq i\leq n+1$ and $f_{n,n+1}=0$ by convention. In a similar fashion, for $1\leq k\leq n,$ we set

$$\begin{equation*} \hat{f}_{k,n+1}=f_{\alpha _k+\cdots +\alpha _{n-1}+\alpha _{n+1}}, \end{equation*}$$

where again we adopt the convention $\hat{f}_{n,n+1}=f_{\alpha _{n+1}}.$ Finally, for $1\leq i < j \leq n-1,$ we set

$$\begin{equation*} F_{i,j}=f_{\alpha _i+\cdots +\alpha _{j-1}+2\alpha _j + \cdots + 2\alpha _{n-1}+\alpha _n+\alpha _{n+1}}, \end{equation*}$$

where $F_{i,i+1} =f_{\alpha _i+2\alpha _{i+1}+\cdots +2\alpha _{n-1}+\alpha _n+\alpha _{n+1}}$, $F_{i,n-1}= f_{\alpha _i+\cdots +\alpha _{n-2}+2\alpha _{n-1} + \alpha _n+\alpha _{n+1}}$ for every $1\leq i\leq n-2.$ We will require the following relations in $\operatorname {Lie}(X).$

Lemma 3.2.

Adopting the notation introduced above, we have

(i)

$f_{\beta _r+\cdots +\beta _s} = \pm f_{r,s}$ for $1\leq r\leq s\leq n-1$,

(ii)

$f_{\beta _r+\cdots +\beta _n}= \pm ( f_{r,n} \pm \hat{f}_{r,n+1})$ for $1\leq r\leq n$,

(iii)

$f_{r,n+1} \in \operatorname {Lie}(X)$ for $1\leq r\leq n-1$,

(iv)

$F_{r,s+1}\in \operatorname {Lie}(X)$ for $1\leq r\leq s\leq n-2$.

Proof.

We start by showing i, arguing by induction on $0\leq s-r \leq n-2.$ If $r=s,$ then the assertion immediately follows from 2, so we assume $1\leq r<s\leq n-1$ in the remainder of the proof. We then successively get

$$\begin{align*} f_{\beta _r+\cdots + \beta _s} &= \pm ( f_{\beta _{r+1}+\cdots +\beta _s}f_{\beta _r} - f_{\beta _r}f_{\beta _{r+1}+\cdots +\beta _s} )\\ &= \pm N_1( f_{r+1,s}f_{\alpha _r} - f_{\alpha _r}f_{r+1,s}) \\ &= \pm N_1( N_2 f_{r,s} + f_{\alpha _r}f_{r+1,s} - f_{\alpha _r}f_{r+1,s} )\\ & = \pm N_1N_2 f_{r,s} \end{align*}$$

for some $N_1,N_2 \in \{\pm 1\},$ where the second equality follows from 2 and our induction assumption. Therefore i holds as desired. For the second assertion, we again argue by induction on $0\leq n-r \leq n-1.$ In the case where $r=n,$ then the result holds by 2, hence we assume $1\leq r\leq n-1$ in the remainder of the proof. We then successively get

$$\begin{align*} f_{\beta _r+\cdots +\beta _n} &= \pm (f_{\beta _{r+1}+\cdots +\beta _n}f_{\beta _r} - f_{\beta _r} f_{\beta _{r+1}+\cdots +\beta _n}) \\ &= \pm N_1 ( f_{r+1,n}f_{\alpha _r} + N_2 \hat{f}_{r+1,n+1}f_{\alpha _r} - f_{\alpha _r}f_{r+1,n} - N_2f_{\alpha _r}\hat{f}_{r+1,n+1} ) \\ &= \pm N_1 ( N_3 f_{r,n} + N_2 \hat{f}_{r+1,n+1}f_{\alpha _r} - N_2f_{\alpha _r}\hat{f}_{r+1,n+1} ) \\ &= \pm N_1 ( N_3 f_{r,n} + N_2N_4 \hat{f}_{r,n+1}) \\ &= \pm N_1N_3 (f_{r,n} + N_2N_3N_4 \hat{f}_{r,n+1}) \end{align*}$$

for some $N_1,N_2,N_3,N_4\in \{\pm 1\},$ where again the second equality follows from 2 and our induction assumption. Therefore ii holds as well. Next we show the third assertion, letting $1\leq r\leq n-1$ be fixed, and setting $\mu =\beta _r+\cdots +\beta _{n-1}+2\beta _n.$ The aforementioned root restrictions yield $\operatorname {Lie}(X)_{\mu }\subset \operatorname {Lie}(Y)_{\alpha _r+\cdots +\alpha _{n+1}},$ and since the latter $T_Y$-weight space is $1$-dimensional, we get that $\operatorname {Lie}(X)_\mu$ is at most $1$-dimensional as well. Now as $f_{\beta _r+\cdots +\beta _n+2\beta _{n+1}}\in \operatorname {Lie}(X)_\mu$ and $\operatorname {Lie}(Y)_{\alpha _r+\cdots +\alpha _{n+1}}=\langle f_{r,n+1}\rangle _K,$ we get that $f_{\beta _r+\cdots +\beta _n+2\beta _{n+1}}$ is a non-zero multiple of $f_{r,n+1},$ from which iii follows. Finally, the assertion iv can be dealt with in a similar fashion.

■

In the remainder of this section, we let $V=L_Y(\lambda )$ be a non-trivial, irreducible $KY$-module having $p$-restricted highest weight $\lambda =\sum _{r=1}^{n+1}{a_r\lambda _r}\in X^+(T_Y),$ and fix a maximal vector $v^+$ in $V$ for $B_Y.$ Setting $\lambda |_{T_X}=\omega ,$ one observes that $v^+$ is a maximal vector of weight $\omega$ in $V$ for $B_X,$ since $B_X\subset B_Y.$ The following result provides a necessary and sufficient condition for $V$ to be irreducible in the case where $a_n\neq 0 =a_{n+1}.$

Lemma 3.3.

Let $V=L_Y(\lambda )$ be as above, and assume $a_n \neq 0=a_{n+1}.$ Then $X$ acts irreducibly on $V$ if and only if $V=\operatorname {Lie}(X)v^+.$

Proof.

First assume $X$ acts irreducibly on $V,$ so that $V|_X\cong L_X(\omega ),$ and observe that since $a_n a_{n+1} =0,$ the $T_X$-weight $\omega$ is $p$-restricted. Therefore the Lie algebra $\operatorname {Lie}(X)$ of $X$ acts irreducibly on $V$ by 10, Theorem 1, from which the desired assertion follows. Conversely, assume $V=\operatorname {Lie}(X)v^+.$ Then $V=\langle Xv^+\rangle$ and hence $V|_X$ has a quotient isomorphic to $L_X(\omega )$ by 16, II, Lemma 2.13 (b). Consequently $V^* |_X$ contains a $KX$-submodule isomorphic to $L_X(\omega ).$ Now $V^*|_X \cong V|_X$ by Lemma 3.1, showing the existence of a submodule $U$ of $V|_X$ such that $U\cong L_X(\omega ).$ Since $(V|_X)_\omega =\langle v^+\rangle _K,$ we get that $v^+\in U$ and so $V=\langle X v^+\rangle \subset U \cong L_X(\omega )$ as desired.

■

In view of Lemma 3.3, a necessary condition for $X$ to act irreducibly on $V=L_Y(\lambda )$, with $\lambda$ such that $a_n\neq 0=a_{n+1}$, is for $f_{\gamma }v^+$ to belong to $\operatorname {Lie}(X)v^+$ for every $\gamma \in \Phi ^+(Y).$ We conclude this section by showing that $V|_X$ is irreducible if and only if $f_{r,n}v^+ \in \operatorname {Lie}(X)v^+$ for every $1\leq r\leq n$ (see Proposition 3.5 below). We first need the following preliminary lemma.

Lemma 3.4.

Let $V=L_Y(\lambda )$ be as above, with $a_n\neq 0=a_{n+1},$ and assume $f_{\gamma }v^+ \in \operatorname {Lie}(X)v^+$ for every $\gamma \in \Phi ^+(Y).$ Then $f_{\gamma }f_{\eta _1}\cdots f_{\eta _s}v^+\in \operatorname {Lie}(X)v^+$ for every $\gamma \in \Phi ^+(Y)$ and $\eta _1,\ldots ,\eta _s \in \Phi ^+(X).$

Proof.

We proceed by induction on $s\geq 1.$ First take $\eta \in \Phi ^+(X)$ and consider $f_\gamma f_\eta v^+,$ with $\gamma \!\in \! \Phi ^+(Y).$ Since $f_\gamma f_\eta v^+ \!=\![f_{\gamma },f_{\eta }]v^+ \!+\! f_{\eta }f_{\gamma }v^+$ and as $f_{\gamma }v^+\!\in \! \operatorname {Lie}(X)v^+$ by assumption, it suffices to show that $[f_{\gamma },f_{\eta }]v^+\in \operatorname {Lie}(X)v^+.$ If $f_\gamma \in \operatorname {Lie}(X),$ then clearly $[f_\gamma ,f_\eta ]v^+$ lies in $\operatorname {Lie}(X)v^+$, so assume $f_\gamma \not \in \operatorname {Lie}(X).$ By Lemma 3.2, we then have that

$$\begin{equation*} \gamma \in \{\alpha _n,\alpha _{n+1}\}\cup \{\alpha _i+\cdots +\alpha _n, \alpha _i+\cdots +\alpha _{n-1}+\alpha _{n+1}\}_{1\leq i\leq n-1}. \end{equation*}$$

Take first $\gamma = \sum _{r=i}^n{\alpha _r}$ for some $1\leq i\leq n,$ so $f_\gamma = f_{i,n},$ and consider $[f_\gamma ,f_\eta ] v^+$. If the latter equals zero, then we immediately get the desired result. So we may assume the contrary and thus we have $f_\eta$ is one of the following:

(i)

$f_{j,i-1}$, $1\leq j\leq i-1.$

(ii)

$f_{\alpha _n}+f_{\alpha _{n+1}}$ (if $i\ne n$).

(iii)

$f_{k,n}\pm \hat{f}_{k,n+1}$, $i+1 \leq k\leq n.$

(iv)

$f_{j,n}\pm \hat{f}_{j,n+1}$, $1\leq j\leq i-1.$

We now calculate $[f_\gamma ,f_\eta ] v^+$ in each case.

(i)

$[f_{i,n},f_{j,i-1}]v^+ = \pm f_{j,n}v^+,$ which lies in $\operatorname {Lie}(X)v^+$ by assumption.

(ii)

$[f_{i,n},f_{\alpha _n}\!+\!f_{\alpha _{n+1}}]v^+ \!=\! \!\pm f_{i,n+1}v^+,$ which lies in $\operatorname {Lie}(X)v^+$ since $f_{i,n+1}\!\in \! \operatorname {Lie}(X)$ by Lemma 3.2iii.

(iii)

$[f_{i,n},f_{k,n}\!\pm \!\hat{f}_{k,n+1}]v^+ \!=\! \pm F_{i,k}v^+,$ which again lies in $\operatorname {Lie}(X)v^+$ as $F_{i,k}\!\in \!\operatorname {Lie}(X)$ by Lemma 3.2iv.

(iv)

$[f_{i,n},f_{j,n}\pm \hat{f}_{j,n+1}]v^+ = \pm F_{j,i}v^+$, which as in the previous case lies in $\operatorname {Lie}(X)v^+.$

Consequently $f_{\gamma }f_{\eta }v^+\in \operatorname {Lie}(X)v^+$ for $\gamma =\sum _{r=i}^{n}{\alpha _r}$ as desired. Arguing in a similar fashion, one shows that the same holds for $\gamma = \sum _{r=i}^{n-1}{\alpha _r}+\alpha _{n+1}$ and $\gamma =\alpha _{n+1}$ as well, where $1\leq i\leq n-1.$ Therefore the lemma holds in the situation where $s=1,$ and hence we assume $s>1$ in the remainder of the proof. For $\eta _1,\ldots ,\eta _s\in \Phi ^+(X),$ we have

$$\begin{equation*} f_{\gamma }f_{\eta _1}\cdots f_{\eta _s}v^+=[f_\gamma ,f_{\eta _1}]f_{\eta _2}\cdots f_{\eta _s}v^+ + f_{\eta _1}f_{\gamma }f_{\eta _2}\cdots f_{\eta _s}v^+. \end{equation*}$$

Now $f_{\gamma }f_{\eta _2}\cdots f_{\eta _s}v^+\in \operatorname {Lie}(X)v^+$ by induction and hence so does $f_{\eta _1}f_{\gamma }f_{\eta _2}\cdots f_{\eta _s}v^+.$ On the other hand, we either have $[f_{\gamma },f_{\eta _1}] =0$ or $[f_{\gamma },f_{\eta _1}] = \xi f_{\delta }$ for some $\xi \in K$ and $\delta \in \Phi ^+(Y)$ by i, ii, iii, iv above, in which case $[f_\gamma ,f_{\eta _1}]f_{\eta _2}\cdots f_{\eta _s}v^+= \xi f_\delta f_{\eta _2}\cdots f_{\eta _s}v^+\in \operatorname {Lie}(X)v^+$ by induction, thus completing the proof.

■

We now establish the following necessary and sufficient condition for $V|_{\operatorname {Lie}(X)}$ to be generated by $v^+$ as a $\operatorname {Lie}(X)$-module (and hence for $V|_X$ to be irreducible by Lemma 3.3), where $V$ has highest weight $\lambda$ with $a_n\neq 0=a_{n+1}.$

Proposition 3.5.

Let $V=L_Y(\lambda )$ be an irreducible $KY$-module having $p$-restricted highest weight $\lambda =\sum _{r=1}^na_r\lambda _r,$ with $a_n \neq 0.$ Then $V=\operatorname {Lie}(X)v^+$ if and only if $f_{r,n}v^+\in \operatorname {Lie}(X)v^+$ for every $1\leq r\leq n.$

Proof.

If $V=\operatorname {Lie}(X)v^+,$ then clearly $f_{\gamma }v^+\in \operatorname {Lie}(X)v^+$ for every $\gamma \in \Phi ^+(Y).$ Hence in particular $f_{r,n}v^+ \in \operatorname {Lie}(X)v^+$ for every $1\leq r\leq n$ as desired. Conversely, suppose that $f_{r,n}v^+ \in \operatorname {Lie}(X)v^+$ for every $1\leq r\leq n$ and notice that since $\lambda \in X^+(T_Y)$ is $p$-restricted, we have

$$\begin{equation*} V=\left\langle f_{\gamma _1}\cdots f_{\gamma _s} v^+ : s\in \mathbb{Z}_{\geq 0},\nobreakspace \gamma _1,\ldots ,\gamma _s\in \Phi ^+(Y)\right\rangle _K \end{equation*}$$

by 10, Theorem 1. Hence in order to show that $V=\operatorname {Lie}(X)v^+,$ it suffices to show that $f_{\gamma _1}\cdots f_{\gamma _s}v^+\in \operatorname {Lie}(X)v^+$ for every $s\in \mathbb{Z}_{>0}$ and $\gamma _1,\ldots ,\gamma _s\in \Phi ^+(Y).$ Assume for a contradiction that this is not the case and let $m\in \mathbb{Z}_{\geq 0}$ be minimal such that there exists $\gamma _1,\ldots ,\gamma _m\in \Phi ^+(Y)$ with $f_{\gamma _1}\cdots f_{\gamma _m}v^+\notin \operatorname {Lie}(X)v^+.$ Lemma 3.2 implies $f_{\gamma }v^+\in \operatorname {Lie}(X)v^+$ for $\gamma \in \Phi ^+(Y)$ and so $m\geq 2.$ Now by minimality, we have $f_{\gamma _2}\cdots f_{\gamma _m}v^+\in \operatorname {Lie}(X)v^+$ and hence an application of Lemma 3.4 completes the proof.

■

3.2. Conclusion

Let $V=L_Y(\lambda )$ be an irreducible $KY$-module having $p$-restricted non-zero highest weight $\lambda =\sum _{r=1}^{n+1}{a_r\lambda _r} \in X^+(T_Y),$ and set $\omega =\lambda |_{T_X}.$ In this section, we will complete the proof of Theorem 1.2. We first show that for certain weights $\lambda ,$ it is straightforward to see that $V|_X$ is reducible. Although the proof of the following proposition can be found in 23, Section 8, we include it here for completeness.

Proposition 3.6.

Let $\lambda \in X^+(T_Y)$ and let $V$ be as above. Then the following assertions hold:

(i)

If $a_n a_{n+1} \neq 0,$ or if $a_n = a_{n+1} =0,$ then $V|_X$ is reducible.

(ii)

If $\lambda \in \{a\lambda _n,\nobreakspace a\lambda _{n+1}: a\in \mathbb{Z}_{> 0} \},$ then $X$ acts irreducibly on $V.$

(iii)

If $V|_X$ is irreducible, so $a_na_{n+1}=0$ by i, and $\lambda$ is not as in ii, then taking $1\leq k <n$ maximal such that $a_k\neq 0,$ we have $p\mid (a_k+a_n+a_{n+1}+n-k).$

Proof.

For i, first consider the case where $a_n a_{n+1} \neq 0.$ Here the $T_X$-weight $\omega '=\omega -\beta _n$ has multiplicity at most $1$ in $L_X(\omega ),$ while each of $\lambda -\alpha _n$ and $\lambda -\alpha _{n+1}$ is a $T_Y$-weight of $V$ restricting to $\omega '.$ Therefore the latter occurs in a second $KX$-composition factor of $V,$ showing that $V|_X$ is reducible as desired. Next assume $a_n = a_{n+1} =0,$ and let $1\leq k\leq n-1$ be maximal such that $a_k \neq 0.$ Then the $T_X$-weight $\omega ''=\omega -(\beta _k+\cdots +\beta _n)$ has multiplicity at most $1$ in $L_X(\omega )$ (since the corresponding weight space is generated by $f_{\beta _k+\cdots +\beta _n}v^+$), while each of $\lambda -(\alpha _k+\cdots +\alpha _n)$ and $\lambda -(\alpha _k+\cdots +\alpha _{n-1}+\alpha _{n+1})$ restricts to $\omega ''.$ Consequently $V|_X$ is reducible in this case as well, and i holds as desired.

Now turn to ii and let $\lambda =a\lambda _n$ for some $a\in \mathbb{Z}_{>0},$ and assume for a contradiction that $V|_X$ is reducible. By 23, 8.5, there exist $1\leq i\leq n$ and a maximal vector $w^+ \in V$ for $B_X$ such that

$$\begin{equation*} 0\neq e_{\alpha _i+\cdots +\alpha _n}w^+ \in \langle v^+ \rangle _K. \end{equation*}$$

In particular $w^+ \in V_{\lambda -(\alpha _i+\cdots + \alpha _n)},$ and one checks that the only $T_Y$-weight of $V$ restricting to $\omega -(\beta _i+\cdots +\beta _n)$ is $\lambda -(\alpha _i+ \cdots + \alpha _n).$ So $w^+ \in V_{\lambda -(\alpha _i+\cdots +\alpha _n)}=\langle f_{\alpha _i+\cdots +\alpha _n}v^+\rangle _K,$ thus yielding $e_{\beta _i}w^+ =e_{\alpha _i}w^+ \neq 0,$ a contradiction. Therefore ii holds as desired.

For iii, we assume $V|_X$ is irreducible, and by i, we suppose, without loss of generality, that $a_n\neq 0 = a_{n+1}.$ Assume as well that $\lambda$ is not as in ii, and seeking a contradiction, take $k$ as in iii, and suppose that $p\nmid (a_k+a_n +n-k).$ Consider the $T_X$-weight $\omega ''=\omega -(\beta _k+\cdots +\beta _n)\in X^+(T_X).$ Then $\lambda -(\alpha _k+\cdots +\alpha _n)$ and $\lambda -(\alpha _k+\cdots +\alpha _{n-1}+\alpha _{n+1})$ are both $T_Y$-weights of $V$ restricting to $\omega ''.$ Now $\operatorname {m}_{L_X(\omega )}(\omega '')\leq \operatorname {m}_{V_X(\omega )}(\omega '') = n-k+1,$ while an application of Proposition 2.2 yields $\operatorname {m}_V(\lambda -(\alpha _k+\cdots +\alpha _n))=n-k+1.$ (Recall that we assumed $a_n\neq 0 = a_{n+1}$ and $p\nmid (a_k + a_n +n-k).$) Now $\operatorname {m}_V(\lambda -(\alpha _k+\cdots +\alpha _{n-1}+\alpha _{n+1}))=1,$ as the latter weight is conjugate to $\lambda -\alpha _k$ under the action of the Weyl group for $Y.$ Therefore $\operatorname {m}_{V|_X}(\omega '') \geq n-k+2 > n-k+1 \geq \operatorname {m}_{L_X(\omega )}(\omega ''),$ yielding the existence of a second composition factor of $V$ for $X,$ a contradiction. The proof is complete.

■

In view of Proposition 3.6, we may and shall assume $\lambda =\sum _{r=1}^n{a_r \lambda _r},$ with $a_n \neq 0$ throughout the rest of the section, as well as the existence of $1\leq k\leq n-1$ maximal such that $a_k \neq 0.$ For $1\leq i\leq n,$ set $P(i,i)=\emptyset$ and for $1\leq i<j\leq n,$ set

$$\begin{equation*} P(i,j)=\left\{(m_r)_{r=1}^s : 1\leq s \leq j-i,\nobreakspace i\leq m_1<\cdots < m_s<j\right\}. \end{equation*}$$

For any sequence $(m)=(m_r)_{r=1}^s\in P(i,j),$ write $f_{(m)}=f_{i,m_1}f_{m_1+1,m_2}\cdots f_{m_s+1,j}.$ By Lemma 2.1, we have that for every $1\leq i<j\leq n,$ the weight space $V_{\lambda -(\alpha _i+\cdots +\alpha _j)}$ is spanned by the vectors

$$\begin{equation*} \{f_{(m)}v^+ : (m)\in P(i,j)\} \cup \{f_{i,j}v^+\}, \end{equation*}$$

where $v^+\in V$ is a maximal vector of weight $\lambda$ for $B_Y.$ We set

$$\begin{equation*} V_{i,j}=\left\langle f_{(m)}v^+:(m)\in P(i,j)\right\rangle _K. \end{equation*}$$

The following special case of 8, Theorem A.7, inspired by 13, Proposition 3.1, shall play a key role in the proof of Theorem 1.2.

Theorem 3.7.

Let $Y$ be a simple algebraic group of type $D_{n+1}$ over $K,$ and consider an irreducible $KY$-module $V=L(\lambda )$ having $p$-restricted highest weight $\lambda =\sum _{r=1}^n{a_r \lambda _r},$ with $a_n\neq 0.$ Then $f_{r,n}v^+\in V_{r,n}$ for every $1\leq r \leq n-1$ if and only if $p\mid (a_i+a_j+j-i)$ for every $1\leq i<j\leq n$ such that $a_ia_j\neq 0$ and $a_s=0$ for $i<s<j.$

Proof.

The result follows from an application of 8, Theorem A.7 to the $A_n$-Levi subgroup of $Y$ corresponding to the simple roots $\alpha _1,\ldots , \alpha _n.$

■

For $1\!\leq \! i\!\leq \! n\!-\!1,$ we let $\hat{V}_{i,n+1}$ denote the $K$-span of $\{f_{i,j}\hat{f}_{j+1,n+1}v^+: i\!\leq \! j\!\leq \! n-1\}$ and $\{f_{(m)}\hat{f}_{j+1,n+1}v^+ : i < j\leq n-1,\nobreakspace (m)\in P(i,j)\}.$ The proof of the main result of this section (namely, Theorem 3.9) relies on the following preliminary result.

Lemma 3.8.

Adopt the notation introduced above and let $1\leq r \leq k-1$ be such that $\hat{f}_{r,n+1}v^+\in \hat{V}_{r,n+1}.$ Then $f_{r,n}v^+\in V_{r,n}.$

Proof.

Let $1\leq r \leq k-1$ be as in the statement of the lemma. If $a_r =0,$ then $f_{\alpha _r}v^+=0$ and hence $f_{r,n}v^+ = \pm f_{\alpha _r}f_{r+1,n}v^+,$ so that $f_{r,n}v^+ \in V_{r,n}$, as claimed. Therefore we assume $a_r\neq 0$ in the remainder of the proof. As $\hat{f}_{r,n+1}v^+\in \hat{V}_{r,n+1}$ and since $\hat{f}_{s,n+1}v^+=0$ for $k<s\leq n,$ we get the existence of $(\xi _s,\xi ^s_{(m)} : r\leq s\leq k-1, (m)\in P(r,s) )\subset K$ such that

$$\begin{equation} \hat{f}_{r,n+1}v^+ = \sum _{s=r}^{k-1}{\left(\sum _{(m)\in P(r,s)}{\xi ^s_{(m)}f_{(m)}\hat{f}_{s+1,n+1}v^+ } + \xi _{s}f_{r,s}\hat{f}_{s+1,n+1}v^+ \right)}. {\tag{3}} \end{equation}$$

Now observe that $f_{\alpha _n} e_{\alpha _{n+1}}\hat{f}_{t,n+1}v^+ \!=\! N_1 f_{t,n}v^+ \!+\! N_2 f_{t,n-1}f_{\alpha _n}v^+$ for every $r\!\leq \! t\!\leq \! k,$ where $N_1,N_2 \in \{\pm 1\}$. Also, for every $r\leq s\leq k-1$ and every $(m)\in P(r,s),$ we have $[e_{\alpha _{n+1}},f_{r,s}]=[e_{\alpha _{n+1}},f_{(m)}]=0,$ as well as $[f_{\alpha _{n}},f_{r,s}]=[f_{\alpha _{n}},f_{(m)}]=0.$ Consequently successively applying $e_{\alpha _{n+1}},$ $f_{\alpha _n}$ to each side of 3 yields $N_1 f_{r,n}v^+ + N_2 f_{r,n-1}f_{\alpha _n}v^+ \in V_{r,n}$ for some $N_1\neq 0,$ $N_2\in \{\pm 1\},$ from which the desired result follows.

■

In the next result, we show that in order to determine whether $V|_X$ is irreducible or not, it is enough to determine whether $f_{r,n}v^+ \in V_{r,n}$ or not, this for every $1\leq r\leq n-1.$

Theorem 3.9.

Let $\lambda =\sum _{r=1}^n{a_r\lambda _r}$ be a $p$-restricted dominant weight such that $a_ka_n\ne 0.$ Let $V=L_Y(\lambda )$ be an irreducible $KY$-module having highest weight $\lambda .$ Then $V|_X$ is irreducible if and only if $f_{r,n}v^+\in V_{r,n}$ for every $1\leq r \leq n-1.$

Proof.

First assume $f_{r,n}v^+\in V_{r,n}$ for every $1\leq r\leq n-1.$ By Proposition 3.5, in order to show that $V|_X$ is irreducible, it suffices to show that $f_{s,n}v^+\in \operatorname {Lie}(X)v^+$ for every $1\leq s\leq n.$ We proceed by induction on $0\leq n-s\leq n-1,$ starting by assuming $s=n.$ Now since $a_{n+1} =0,$ we immediately get that $f_{\alpha _{n+1}}v^+=0$ and hence $f_{\beta _n}v^+=f_{\alpha _n}v^+$ by 2, that is, $f_{\alpha _n}v^+\in \operatorname {Lie}(X)v^+.$ Next assume $1\leq s\leq n-1.$ Since $f_{s,n}v^+\in V_{s,n}$ by assumption, there exists $(\xi _i, \xi ^i_{(m)}: s\leq i\leq n-1, (m)\in P(s,i))\subset K$ such that

$$\begin{equation*} f_{s,n}v^+ = \sum _{i=s}^{n-1} {\left( \xi _{i}f_{s,i}f_{i+1,n}v^+ +\sum _{(m)\in P(s,i)}{ \xi ^i_{(m)}f_{(m)} f_{i+1,n}v^+}\right)}. \end{equation*}$$

By Lemma 3.2, $f_{s,i}\in \operatorname {Lie}(X)$ and $f_{(m)} \in \operatorname {Lie}(X)$ for every $s\leq i \leq n-1$ and every $(m)\in P(s,i).$ Furthermore, we also have $f_{i+1,n}v^+\in \operatorname {Lie}(X)v^+$ for $s\leq i \leq n-1$ by induction. Therefore $f_{s,n}v^+\in \operatorname {Lie}(X)v^+$ as desired.

Conversely, assume $V|_X$ is irreducible, and let $1\leq r\leq n-1$ be fixed. If $a_r =0,$ then $f_{\alpha _r}v^+=0$ and hence $f_{r,n}v^+ = \pm f_{\alpha _r}f_{r+1,n}v^+,$ so that $f_{r,n}v^+ \in V_{r,n}$ in this situation. In addition, observe that $p\mid (a_k+a_n+n-k)$ by Proposition 3.6(iii), thus yielding $f_{k,n}v^+\in V_{k,n}$ by Proposition 2.2. Therefore we assume $a_r\neq 0$ in the remainder of the proof, and $r<k.$ By Lemma 3.3, we have $V=\operatorname {Lie}(X)v^+,$ in which case Proposition 3.5 applies, thus yielding $f_{r,n}v^+\in \operatorname {Lie}(X)v^+.$ Since $f_{r,n}v^+\in (\operatorname {Lie}(X)v^+)_{\omega -(\beta _r+\cdots +\beta _n)},$ we get the existence of $\xi \in K,$ $x\in V_{r,n}$, and $y\in \hat{V}_{r,n+1}$ such that $f_{r,n}v^+ = \xi f_{r,n}v^+ \pm \xi \hat{f}_{r,n+1}v^+ + x + y.$ Comparing $T_Y$-weights yields

$$\begin{equation*} (\xi -1)f_{r,n}v^+ \in V_{r,n} \text{ and } \pm \xi \hat{f}_{r,n+1}v^+ \in \hat{V}_{r,n+1}. \end{equation*}$$

If $\xi \neq 1,$ then the assertion is immediate, while if on the other hand $\xi =1,$ then an application of Lemma 3.8 yields the desired result.

■

We are now able to complete the proof of Theorem 1.2.

Proof of Theorem 1.2.

First assume $X$ acts irreducibly on $V.$ By Proposition 3.6(i), we then have that, up to a graph automorphism of $Y$, $a_n\neq 0=a_{n+1}$, and if $\lambda =a\lambda _n$, then $\lambda$ is as in the statement of the result. So assume the existence of $1\leq k\leq n-1$ maximal with $a_k\ne 0$. Then Theorem 3.9 applies, yielding $f_{r,n}v^+\in V_{r,n}$ for every $1\leq r\leq n-1.$ An application of Theorem 3.7 then implies the desired divisibility conditions.

Conversely, assume $\lambda \in X^+(T_Y)$ satisfies the conditions in Theorem 1.2. In the case where $\lambda =a\lambda _n$ for some $a\in \mathbb{Z}_{>0},$ then the result follows from Proposition 3.6(ii), hence we assume the existence of $1\leq k\leq n-1$ maximal such that $a_ka_n\neq 0,$ and assume moreover the divisibility conditions as in the theorem. Here an application of Theorem 3.7 yields $f_{r,n}v^+\in V_{r,n}$ for every $1\leq r \leq n-1,$ and hence Theorem 3.9 then shows that $X$ acts irreducibly on $V$ as desired.

■

4. Proof of Propositions 1.5 and 1.7

We first prove Proposition 1.5. Let $Y=\operatorname {Spin}_{2n+2}(K)$ be a simply connected, simple algebraic group of type $D_{n+1}$ over $K,$ with $n\geq 2.$ Let $X\subset Y$ be a semisimple, connected, proper, closed subgroup of $Y.$ Fix $T_Y$ a maximal torus of $Y$ and let $T_Y \subset B_Y$ denote a Borel subgroup of $Y.$ Also let $\{\lambda _1,\ldots ,\lambda _{n+1}\}$ be the corresponding set of fundamental dominant weights for $T_Y.$ In this section, we give a proof of Proposition 1.5, starting with three results, proven in a more general setting in 23, Section 5. For the convenience of the reader, and in order to render the current manuscript more self-contained, we include the proofs of these special cases here.

Proposition 4.1.

Let $Y$ be a simply connected, simple algebraic group of type $D_{n+1}$ with natural module $W$ and let $X$ be a semisimple, connected, proper, closed subgroup of $Y$ acting irreducibly on a non-trivial, $p$-restricted, irreducible $KY$-module $V=L_Y(\lambda )$. Then one of the following holds:

(i)

$W|_X$ is irreducible.

(ii)

$W|X$ is reducible and $X\subset B_k\times B_{n-k}$ for some $0\leq k<n$.

Proof.

Suppose $W|_X$ is reducible. Let $W_1$ be a minimal non-zero $X$-invariant subspace of $W$. Then $W_1\cap W_1^\perp = W_1$ or $\{0\}$. Suppose $W_1\cap W_1^\perp = W_1$. Note that $W_1$ is not totally singular, else $X$ lies in a proper parabolic subgroup of $Y$ and so cannot act irreducibly on $V$. Therefore, we have $p=2$; the set of singular vectors in $W_1$ being an $X$-invariant subspace of $W_1$ forces $W_1$ to be generated by a non-singular vector, and so ii holds with $k=0$. In case $W_1\cap W_1^\perp =\{0\}$, set $W_2 = W_1^\perp$, so that $W=W_1\oplus W_2$, an orthogonal direct sum and the image of $X$ in $\operatorname {Isom}(W)$ lies in $\operatorname {Isom}(W_1)'\times \operatorname {Isom}(W_2)'$. If $\dim W_1$ is even, then $X\subset D_s\times D_{n+1-s}$, a maximal rank subgroup of $Y$ and we may invoke 23, Theorem 4.1 to see that $X$ acts reducibly on $V$. So $\dim W_1$ is odd and $X\subset B_k\times B_{n-k}$ as in ii.

■

Proposition 4.2.

Let $Y$ and $V$ be as in Proposition 4.1(ii) for some $0<k<n$, and let $V$ be of highest weight $\lambda$. Then the closed, connected subgroup $H\subset Y$ of type $B_k\times B_{n-k}$ acts irreducibly on $V$ if and only if $\lambda = \lambda _n$ or $\lambda _{n+1}$.

Proof.

To see that $H$ acts irreducibly on the two half-spin modules for $Y$, we simply compute the restriction of the highest weight to a maximal torus of $H,$ note that the restriction induces the tensor product of the spin modules for the two simple factors of $H,$ and then a dimension comparison completes the proof.

We now assume $H$ acts irreducibly on $V$, the irreducible $KY$-module of highest weight $\lambda$. We proceed by induction on $n$ and first consider the case $n=2,$ so that $Y$ is of type $D_3 = A_3$ and the image of $H$ in $\operatorname {SO}_6$ lies in the subgroup $\operatorname {SO}_3\times \operatorname {SO}_3\subset \operatorname {SO}_6$. Since $H$ acts irreducibly on $V$, $H$ does not lie in a parabolic subgroup of $Y$ and so acts irreducibly on the $4$-dimensional $p$-restricted $KY$-modules, which correspond to the highest weights as in the statement of the result. Thus the preimage of $\operatorname {SO}_3\times \operatorname {SO}_3$ in $\operatorname {SL}_4$ acts on the 4-dimensional module via the tensor product representation of $A_1\times A_1$ on $E\otimes E$, where $E$ is the natural $2$-dimensional representation of $\operatorname {SL}_2$. To see that $\operatorname {SL}_2\times \operatorname {SL}_2$ acts reducibly (and hence $H$ as well) on all other non-trivial, $p$-restricted, irreducibles for $Y,$ we require a further argument (and some additional notation).

Let $T$ be a maximal torus of $\operatorname {SL}_2\times \operatorname {SL}_2$ with $T\subset T_Y.$ Moreover, choose a base $\Pi = \{\alpha ,\beta \}$ of the root system of $\operatorname {SL}_2\times \operatorname {SL}_2$, and a base $\Pi (Y)=\{\alpha _1,\alpha _2,\alpha _3\}$ of the root system of $Y$, viewing $Y$ as $D_3$. Now it is straightforward to see that up to conjugation we may assume that $\alpha _1|_T = \beta -\alpha$, $\alpha _j|_{T} = \alpha$ for $j=2,3$. Now we apply 23, 6.1 to see that $\lambda \in \{c\lambda _j,b\lambda _1+a\lambda _j, c\geq 1, b>0, a+b=p-1, j=2,3\}$. In case $\lambda =c\lambda _j$, $V|_{\operatorname {SL}_2\times \operatorname {SL}_2} = S^c(E\otimes E)$ which is easily seen to be reducible if $c>1$. In any case, $V|_{\operatorname {SL}_2\times \operatorname {SL}_2}$ has a composition factor with highest weight $\lambda |_T$. In the cases $\lambda =b\lambda _1+a\lambda _j$, we see that the weight $\lambda -\alpha _1$ restricts to $T$ as $\lambda |_T-\beta +\alpha$ and hence lies in a second composition factor of $V|_{\operatorname {SL}_2\times \operatorname {SL}_2}$. The result then holds for $n=2$.

Assume now that $n\geq 3$ and that the result holds for $Y=D_\ell$ with $\ell <n+1$. Assume as well that $p>2$; we will treat the case $p=2$ at the end of the proof. Let $W_1$ and $W_2$ be as in the previous proof so that $B_k$ acts on $W_1$ and $B_{n-k}$ acts on $W_2$. Let $U_1$, respectively $U_2$, be maximal totally isotropic subspaces of $W_1$, respectively $W_2$. Then $U_0=U_1\oplus U_2$ is an $n$-dimensional totally isotropic subspace of $W$. Let $P\subset Y$ be the preimage of the stabilizer in $\operatorname {Isom}(W_1)'\times \operatorname {Isom}(W_2)'$ of $U_0$ and $R$ the preimage in $Y$ of the stabilizer in $\operatorname {Isom}(W)$ of $U_0$. Then $P\subset R$, $R_u(P)\subset R_u(R)$, and $(P/R_u(P))'\cong \operatorname {SL}(U_1)\times \operatorname {SL}(U_2)$, while $(R/R_u(R))'\cong \operatorname {SL}(U_0)=\operatorname {SL}_n$. In particular, the image of the Levi factor of $P$ in $R/R_u(R)$ stabilizes $U_1$ and so lies in a proper parabolic subgroup of $R/R_u(R)$, and hence can act irreducibly on no non-trivial $(R/R_u(R))'$-module. Then the above remarks and an application of the main proposition of 24 shows that $\lambda =a\lambda _n+b\lambda _{n+1}$ for some $a,b$.

Assume $\dim W_1\geq \dim W_2$, so $\dim W_1\geq 5$. (Recall that $\dim V \geq 8.$) Let $P_1$ be the stabilizer in $\operatorname {SO}(W_1)'$ of a singular $1$-space. Then $P_1\times \operatorname {SO}(W_2)'$ is a proper parabolic subgroup of $\operatorname {SO}(W_1)'\times \operatorname {SO}(W_2)'$ and is contained in the image (under the natural projection $Y\rightarrow \operatorname {SO}(W)$) of the stabilizer in $Y$ of this $1$-space. As $P_1 = B_{k-1} R_u(P_1)$, we have the Levi factor $B_{k-1}B_{n-k}$ projecting into the Levi factor of type $D_n$. Another application of 24 and the induction hypothesis yield the result.

Turn now to the case $n>2$ and $p=2.$ In this case we have that the subgroup $B_k \times B_{n-k}$ stabilizes a non-singular 1-space and so lies in a subgroup of type $B_n$. Now the subgroup $B_{k}\times B_{n-k}$ is a maximal rank subgroup of $B_n$ and we can appeal to 23, Theorem 4.1 applied to the pair $(B_k\times B_{n-k},B_n)$ to see that the irreducible $KB_n$-module $V|_{B_n}$ is a twist of the spin module and hence we deduce that $\lambda =\lambda _n+\lambda _{n+1},$ $\lambda _n,$ or $\lambda _{n+1}.$ However, $L_Y(\lambda _n+\lambda _{n+1})|_{B_{n}}$ is not irreducible, as the highest weight affords a twist of the spin module for $B_n$ and this is of dimension strictly less than the dimension of $L_Y(\lambda _n+\lambda _{n+1})$. So $\lambda =\lambda _n$ or $\lambda _{n+1}$ as claimed.

■

Propositions 4.1 and 4.2 show that under the hypotheses of Proposition 1.5, either Proposition 1.5(a) holds, or $X$ acts irreducibly on the natural $KY$-module $W$, or $X\subset B_n$ is as in Proposition 4.1(ii) with $k=0$. Theorem 1.2 handles the latter situation in case $X=B_n$. The resolution of this case will follow from induction; see the end of this section. Now for the case where $X$ acts irreducibly on $W$, we first show that $X$ must act tensor indecomposably.

Proposition 4.3.

Let $Y,$ $X,$ $V$ be as in Proposition 1.5, and let $W$ be the natural $(2n+2)$-dimensional $KY$-module. If $W|_X$ is irreducible, then $W|_X$ is tensor indecomposable.

Proof.

Suppose the contrary. Then write $W|_X=W_1^{(f_1)}\otimes \cdots \otimes W_t^{(f_t)}$, where $W_i$ is a $p$-restricted $KX$-module, $f_1,\dots ,f_t$ are distinct $p$-powers, and $t\geq 2$. Then the criterion on the highest weights of self-dual modules shows that each $W_i$ carries an $X$-invariant non-degenerate bilinear form. Write $W=D\otimes F$, where each of $D$ and $F$ is a $KX$-module and the image of $X$ in $\operatorname {Isom}(W)'$ lies in $\operatorname {Isom}(D)'\circ \operatorname {Isom}(F)'$, where this latter denotes the image of the natural morphism ${\mathrm{Isom}}(D)'\times {\mathrm{Isom}}(F)'\rightarrow {\mathrm{Isom}}(W)$, a commuting product of subgroups isomorphic to ${\mathrm{Isom}}(D)'$ and ${\mathrm{Isom}}(F)'$. Set $\dim D=d$ and $\dim F=f$ and assume $d\geq f\geq 2$ (and so at least one of $d$ and $f$ is even).

Extracting part of the argument given on page 76 of 23, we will now show that $Y$ contains a semisimple group inducing $\operatorname {Isom}(D)'\circ \operatorname {Isom}(F)'$. As pointed out above, $X$ stabilizes the product bilinear form on $W$. But then the irreducibility of $X$ on $W$ forces this to be a scalar multiple of the form defining $Y$. Adjusting the form on $D$ if necessary, we may assume that the product form is precisely the form defining $Y$. This establishes the claim unless $p=2$. In this case, we have $\operatorname {Isom}(D)'\circ \operatorname {Isom}(F)'\subset \operatorname {Sp}(D)\circ \operatorname {Sp}(F)$. The latter group preserves a quadratic form $\mathcal{Q}$ on $W$, such that $\mathcal{Q}$ has the same polarization as the bilinear form on $W$ and also such that $\mathcal{Q}(x\otimes y)=0$ for all $x\in D$ and $y\in F$ (see 17, Section 4.4). By 2, 4.9, $X$ fixes a unique quadratic form with prescribed bilinear form, and so this must necessarily be the form $\mathcal{Q}$ and we again have the claim. Hence we now have that $X$ lies in a closed subgroup $J$ of $Y$ which is the preimage in $Y$ of the group $\operatorname {Isom}(D)'\circ \operatorname {Isom}(F)'$. In particular $J$ acts irreducibly on $V$ as well. Let $D_1$ be an isotropic $1$-space in $D$; then $D=D_1\oplus D_2\oplus D_3$, where $D_1^\perp =D_1\oplus D_2$, $D_2$ is non-degenerate, and $D_3$ is an isotropic $1$-space. Similarly decompose $F=F_1\oplus F_2\oplus F_3$. Now consider the flag of subspaces in $V$:

$$\begin{equation*} 0=V_0\subset V_1\subseteq V_2\subseteq V_3\subseteq V_4\subset V_5=V, \end{equation*}$$

where $V_1=D_1\otimes F_1$, $V_2=(D_1\otimes F_1^\perp )+(D_1^\perp \otimes F_1)$, $V_3=V_2^\perp$, and $V_4=V_1^\perp$. Then $V_1$ and $V_2$ are totally isotropic (totally singular if $p=2$), $\dim V_1=1=\dim (V/V_4)$, $\dim (V_2/V_1)=\dim (V_4/V_3)=d+f-4$, and $\dim (V_3/V_2)=(d-2)(f-2)+2$.

Set $P_J$ to be the stabilizer in $J$ of the above flag, and $P_Y$ the stabilizer in $Y$ of this flag. Then $P_J$ is the product of the preimages of the parabolic subgroups $P_D$ and $P_F$ of $\operatorname {Isom}(D)$ and $\operatorname {Isom}(F)$ which are the stabilizers of the isotropic $1$-spaces $D_1$ and $F_1$, respectively. So $R_u(P_D)$ acts trivially on $D_1$, $D_1^\perp /D_1$, and $D/D_1^\perp$, and similarly for $R_u(P_F)$. On the other hand, $R_u(P_Y)$ is precisely the subgroup of $Y$ which acts trivially on $V_{j+1}/V_j$ for $0\leq j\leq 4$. One then sees that $R_u(P_J)\subset R_u(P_Y)$. Now $(P_Y/R_u(P_Y))'\cong (\operatorname {Isom}(V_3/V_2))'\circ \operatorname {SL}(V_2/V_1)$ and $(P_J/R_u(P_J))'\cong (\operatorname {Isom}(D_1^\perp /D_1))'\circ (\operatorname {Isom}(F_1^\perp /F_1))'$.

Assume for the moment that $f>2$; then $V_3/V_2$ is an orthogonal space of dimension at least 4. We note that the subspace $((D_1\otimes F_3) + V_2)/V_2$ is a non-zero singular subspace in $V_3/V_2$ left invariant by $(\operatorname {Isom}(D_1^\perp /D_1))'\circ (\operatorname {Isom}(F_1^\perp /F_1))'$ and so the projection of this latter group in $\operatorname {Isom}(V_3/V_2)$ is contained in a proper parabolic. It then follows from an application of 24 to the irreducible $KY$-module $V$ and the subgroup $J$ that the portion of the Dynkin diagram for $Y$ corresponding to the subgroup $\operatorname {Isom}(V_3/V_2)$ has zero labels, when representing $\lambda$ by a labeled diagram. But this contradicts our assumption on $\lambda$.

Consider now the case where $f=2$, so $\dim W=2n+2$ and $d=n+1$. Since we are assuming $Y=D_{n+1}$ with $n\geq 4$, we have $d\geq 5$. Now $\operatorname {Isom}(D_2)\circ \operatorname {Isom}(F_2)$ stabilizes the image of $D_1\otimes F_1^\perp$ in $V_2/V_1$ and so lies in a proper parabolic subgroup of $\operatorname {Isom}(V_2/V_1)$. Note that $\dim (V_2/V_1) = n-1$, and arguing as above we deduce that the nodes corresponding to $\operatorname {SL}(V_2/V_1)$ in the Dynkin diagram are labelled zero. So now we have $\lambda =a_1\lambda _1+a_n\lambda _n$, with $a_n\ne 0$. Moreover, the image of $X$ in $\operatorname {Isom}(W)'$ lies in $\operatorname {Isom}(D)'\circ \operatorname {Isom}(F)'= \operatorname {Isom}(D)'\circ \operatorname {Sp}_2$, which then implies that $D$ is even-dimensional and the latter group is $\operatorname {Sp}(D)\circ \operatorname {Sp}_2$, and acts irreducibly on $V$. The factor $\operatorname {Sp}(D)$ lies in the derived subgroup of an $A_n$-type Levi factor $L$ of $Y$, indeed is the naturally embedded $\operatorname {Sp}_{n+1}$ subgroup of $A_n$. Moreover, $\operatorname {Sp}(D)$ acts homogeneously on $V$. Now $\lambda$ and $\lambda -\alpha _n$, or $\lambda -\alpha _{n-1}-\alpha _n-\alpha _{n+1}$, depending on whether the root system of $L$ contains $\alpha _{n+1}$ or $\alpha _n$, afford the highest weights of irreducible summands of $V|_L$. It is then straightforward to see that the restrictions of these weights to the subgroup $\operatorname {Sp}(D)$ provide non-isomorphic composition factors. This provides the final contradiction in case $f=2$ and completes the proof of the result.

■

For the proof of Proposition 1.5, we continue with our consideration of the case where $X$ acts irreducibly and, by the previous result, tensor-indecomposably on the natural $KY$-module $W$. In particular, we may now assume that $X$ is a simple, proper, closed subgroup of $Y$. The hypotheses of Proposition 1.5 then imply that $X$ is of type $B_m$ or of type $F_4$. Moreover, we have the full set of hypotheses on the embedding of a parabolic subgroup of $X$ in a parabolic subgroup of $Y$, in particular, with respect to the restriction of the highest weight $\lambda$ to a Levi factor. We will in fact show that the irreducibility of $W|_X$ and the hypotheses of Proposition 1.5 are incompatible.

Taking $X$ a simple counterexample of minimal rank, we see that we may assume $P_X$ is a maximal parabolic subgroup of $X$. In each case, we will require some detailed information about the commutator series of an irreducible $KX$-module with respect to a fixed maximal parabolic subgroup. We start by considering:

Case 1: $X$ of type $B_m$, $m\geq 4$

Fix a maximal torus $T_X$ of $X$. Let $\Pi (X)=\{\beta _1,\dots ,\beta _m\}$ be a base of the root system of $X$, $B_X$ the corresponding Borel subgroup containing $T_X$, and $\{\omega _1,\dots ,\omega _m\}$ a set of fundamental dominant weights chosen with respect to the fixed base $\Pi (X)$. Set $s_i$ to be the reflection corresponding to the root $\beta _i$ for $1\leq i\leq m.$ For a torus $S$ of $X$, write $X(S)$ for the group of rational characters of $S$. Let $P_X$ be a maximal parabolic subgroup of $X$ corresponding to the subset $\{\beta _i : 2\leq i \leq m\}$, and containing the opposite Borel subgroup $B_X^-$. Let $Q_X=R_u(P_X)$ and $P_X=L_XQ_X$ for a Levi factor $L_X$ of $P_X.$ Let $\omega =\sum _{i=1}^m d_i\omega _i$ be a $p$-restricted dominant weight in $X(T_X)$, set $M=L_X(\omega )$ and assume $X$ preserves a non-degenerate quadratic form on $M$, and let us denote this by $\mathcal{Q}$ and the associated bilinear form by $(\ ,\ ):M\times M\rightarrow K$.

For a unipotent group $J$ and a $KJ$-module $N$, we recall the standard notation $[N,J]$ for the subspace spanned by the set of vectors $n-xn$, where $x\in J$ and $n\in N$. We introduce an additional notation: set $[N,J^0]=N$ and set $[N,J^{i+1}]=[[N,J^i],J]$ for $i\geq 0$, so $[N,J^1]=[N,J]$.

Definition 4.4.

Let $M=L_X(\omega )$ be as above.

(i)

We will say a $T_X$-weight $\nu$ in $M$ has level $i$ if $\nu =\omega -i\beta _1-\sum _{j=2}^m c_j\beta _j$ for some integers $i,c_j\geq 0$.

(ii)

Let $e(\omega )$ denote the maximum level of a weight. (When $\omega$ is fixed, we will simply write $e$.)

Lemma 4.5.

Let $M = L_X(\omega )$. Then the following assertions hold:

(i)

For $i\geq 0$, $[M,Q_X^i] = \sum M_\nu$, where the sum ranges over all $T_X$-weights $\nu$ of level at least $i.$

(ii)

For $i\geq 0$, the $L_X'$-module $[M,Q_X^i]/[M,Q_X^{i+1}]$ is isomorphic to the module$$\begin{equation*} \sum _{\nu \in X(T_X)\text{ of level } i}M_\nu . \end{equation*}$$

(iii)

The maximum level of weights in $M$ is $e=2(\sum _{i=1}^{m-1}d_i)+d_m$. If a weight $\nu$ is of level $j$, then $-\nu$ is of level $e-j$.

(iv)

Let $S$ be a subtorus of $T_X$. For weights $\eta ,\chi \in X(S)$, with $\eta \ne -\chi$, we have $V_\eta \subseteq V_\chi ^\perp$. Moreover, if $\eta \ne 0$, then ${\mathcal{Q}}(u)=0$ for all $u\in V_\eta$.

(v)

The subspace $[M,Q_X^i]$ is totally singular for $i\geq (e+1)/2$.

(vi)

If $e$ is even, then the subspace$$\begin{equation*} \sum _{\nu \in X(T_X)\text{ of level } e/2} M_\nu \end{equation*}$$

is non-degenerate.

(vii)

For $i\geq (e+1)/2,$ we have $[M,Q_X^i]^\perp = [M,Q_X^{e-i+1}].$

(viii)

For all $i\geq (e/2)$, the quadratic form on $M$ induces an $L_X'$-invariant quadratic form on $[M,Q_X^i]/[M,Q_X^{i+1}]$. If the form is non-degenerate on this quotient space, then there exists $\nu$ of level $i$ such that $-\nu$ is also of level $i$. In particular, in this case, $d_m$ is even.

Proof.

We start by proving i, arguing by induction on $i,$ the case $i=0$ being trivial by definition. Let $i\geq 1,$ and fix an ordering on $\Phi ^+(X)$ such that any root having a non-zero coefficient of $\beta _1$ when expressed in terms of the simple roots is smaller than the others. By 15, Section 27, we clearly have that $[M,Q_X^i]\subseteq \sum M_\nu ,$ where the sum ranges over all $T_X$-weights $\nu$ of level at least $i,$ and so it remains to show that $M_\nu \subseteq [M,Q_X^i]$ for all such $T_X$-weights $\nu .$ It is enough to show the latter for $T_X$-weights of level exactly $i.$ Let then $\nu$ be a $T_X$-weight of level $i$ in $M,$ and let $\gamma _1 \leq \cdots \leq \gamma _k$ be roots in $\Phi ^+(X)$ such that $\sum _{r=1}^k{\gamma _r}=\lambda -\nu .$ We show that $f_{\gamma _1}\cdots f_{\gamma _k}v^+ \in [M,Q_X^i],$ where $v^+$ is a maximal vector in $M.$ (This will be enough by Lemma 2.1.) Set $w=f_{\gamma _2}\cdots f_{\gamma _k}v^+.$ Then $w$ has level $i-1,$ as $\gamma _1$ involves $\beta _1$ thanks to our choice of ordering on $\Phi ^+,$ and hence $w\in [M,Q_X^{i-1}]$ by induction. Also, we have

$$\begin{equation*} x_{-\gamma _1}(1)w - w \in f_{\gamma _1}\cdots f_{\gamma _k}v^+ + \sum _{r=1}^{\infty }{M_{\nu -r\gamma _1}}, \end{equation*}$$

and hence $x_{-\gamma _1}(1)w - w$ has non-zero coefficient of $f_{\gamma _1}\cdots f_{\gamma _k}v^+,$ and all other terms in the sum are weight vectors of weights different from $\nu .$ One then deduces that $[M,Q_X^i]$ must contain $f_{\gamma _1}\cdots f_{\gamma _k}v^+$ as desired, showing that i holds.

The statement of ii now follows by induction on $i$.

For iii, set $w_0$ to be the longest word of the Weyl group of $X$. Writing the $\omega _i$ in terms of the simple roots $\beta _j$, we see that

$$\begin{equation*} w_0(\omega ) = -\omega =\omega -2(d_1+d_2+\cdots +d_{m-1} + \frac{1}{2}d_m)\beta _1-\sum _{i=2}^m c_i\beta _i \end{equation*}$$

for some non-negative integers $c_i$, giving the result. For the final statement, suppose that $\nu =\omega -j\beta _1-\sum _{i=2}^m c_i\beta _i$, then $-\nu =\omega -2\omega +j\beta _1+\sum _{i=2}^mc_i\beta _i = \omega -(e-j)\beta _1-\sum _{i=2}^mb_i\beta _i$ for some non-negative integers $b_i$.

The statement of iv is standard, and v and vi follow from iv.

For vii, note that by i and iv, we have $[M,Q_X^{e-i+1}]\subseteq [M,Q_X^i]^\perp$. Then the result follows from a dimension argument using ii and iii.

For viii, we note that for $i\geq (e/2)$, $[M,Q_X^{i+1}]$ is totally singular and so the given quadratic form induces an $L_X'$-invariant form on the quotient. Now if $p\ne 2$, the second statement follows directly from iv. If $p=2$ and $[M,Q_X^i]/[M,Q_X^{i+1}]$ is odd-dimensional, so that the bilinear form has a 1-dimensional non-singular radical, then iv implies that the radical is generated by a vector of weight 0. So the weight $\nu =0$ satisfies the given condition. For the claim about the parity of $d_m$, we recall that for $\nu$ of level $j$, $-\nu$ is of level $e-j$. So we deduce that $e$ is even, and so $d_m$ as well.

■

In case $e$ is even, we will consider a certain $T_X$-weight at level $(e/2)$, namely the weight

$$\begin{equation*} \mu : = \omega -(e/2)\beta _1-(e/2-d_1)\beta _2 - \cdots - (d_{m-1}+(d_m/2))\beta _{m-1}-(d_m/2)\beta _m. \end{equation*}$$

Note that

$$\begin{equation} \mu = -(e/2)\omega _1+\sum _{i=2}^{m-1}d_{i-1}\omega _i+(2d_{m-1}+d_m)\omega _m. {\tag{4}} \end{equation}$$

Lemma 4.6.

Assume $d_m$ is even. For each $0\leq j\leq d_m$, there exists a unique weight $\nu _j\in X(T_X)$, of level $j+\sum _{i=1}^{m-1}d_i$, such that $\nu _j|_{T_X\cap L_X'} = \mu |_{T_X\cap L_X'}$. In addition, each $\nu _j$ is of multiplicity $1$ and if $\nu \in X(T_X)$ with $\nu |_{T_X\cap L_X'} = \mu |_{T_X\cap L_X'}$, then $\nu =\nu _j$ for some $j$. In particular, there exists an odd number of weights $\nu$ such that $\nu |_{T_X\cap L_X'} = \mu |_{T_X\cap L_X'}$.

Proof.

Let $\nu \in X(T_X)$ such that $\nu |_{T_X\cap L_X'} = \mu |_{(T_X\cap L_X')}$. Then $\nu =\omega -\sum _{i=1}^m c_i\beta _i,$ and the $c_i$ must satisfy the equations

$$\begin{equation*} d_{j+1}+c_j-2c_{j+1}+c_{j+2} = d_j\text{ for }1\leq j\leq m-2,\text{ and} \end{equation*}$$

$$\begin{equation*} d_m+2c_{m-1}-2c_m = 2d_{m-1}+d_m. \end{equation*}$$ Solving these equations leads to the relations $c_j=c_m+\sum _{i=j}^{m-1}d_i$ for $1\leq j\leq m-1$. Now note as well that for $\nu$ as given, $s_{m-1}s_{m-2}\cdots s_{1}\nu = \omega -c_m\beta _m$. Hence $\nu$ has multiplicity at most 1 and has multiplicity 1 if and only if $c_m\leq d_m$. So for all $0\leq j\leq d_m$, there exists a weight $\nu _j$ of multiplicity $1$ and of level $j+\sum _{i=1}^{m-1}d_i$ whose restriction to $T_X\cap L_X'$ is equal to $\mu |_{(T_X\cap L_X')}$.

■

Proposition 4.7.

Assume the hypotheses of Proposition 1.5 with $X=B_m$. Then $X$ acts reducibly on the natural $KY$-module.

Proof.

Suppose the contrary, that is, let $W$ be the natural $KY$-module and suppose that $W|_X$ is irreducible. Then by Proposition 4.3, $W|_X$ is tensor indecomposable. If $W|_X$ is not $p$-restricted, then we replace $W|_X$ by a $p$-restricted representation $\rho$ of a simply-connected cover $\tilde{X}$ of $X$, and consider $W|_{\rho (\tilde{X})}$. For the purposes of the argument, we may replace $X$ by $\rho (\tilde{X})$. Let $W=L_X(\omega )$ for $\omega =\sum _{i=1}^md_i\omega _i$, a $p$-restricted dominant weight.

As in the hypotheses of Proposition 1.5, let $P_Y$ be a parabolic subgroup of $Y$ containing $P_X$ with $Q_X\subset R_u(P_Y) = Q_Y$. Fix a maximal torus $T_Y$ of $P_Y$ with $T_X\subset T_Y$. We choose a base $\Pi (Y)=\{\alpha _1,\dots ,\alpha _{n+1}\}$ of the root system of $Y$ so that $P_Y$ contains the opposite Borel $B_Y^-$ subgroup with respect to this base.

Now let $\{0\}=W_0\subsetneq W_1\subsetneq \cdots \subsetneq W_{t-1}$ be a flag of totally singular subspaces, such that $P_Y$ is the full stabilizer in $Y$ of this flag; so $P_Y$ is the stabilizer of the flag

$$\begin{equation*} \{0\}=W_0\subsetneq \cdots \subsetneq W_{t-1}\subseteq W_{t-1}^\perp \subsetneq W_{t-2}^\perp \cdots \subsetneq W_0^\perp =W. \end{equation*}$$

Setting $W_{t-1+j}=W_{t-j}^\perp$ for $0\leq j\leq t,$ we have the flag

$$\begin{equation*} \{0\} = W_0\subsetneq \cdots \subsetneq W_{t-1}\subseteq W_t\subsetneq W_{t+1} \cdots \subsetneq W_{2t-1}=W, \end{equation*}$$

of which $P_Y$ is the full stabilizer. By our hypotheses on $P_Y$, we have that $W_t/W_{t-1}$ is an orthogonal space of dimension at least 8 (so $W_{t-1}\ne W_t$) and we have $L_t=(\operatorname {Isom}(W_t/W_{t-1}))'$, a group of type $D_m$ for $m\geq 4$. Note that $W_j = [W,Q_Y^{2t-1-j}]$ for all $j\geq 0$.

Recall that by hypothesis, $L_X'$ is of type $B_{m-1}$. If $m>4$, $L_X'$ is the stablizer in $L_t$ of a non-singular $1$-space of $W_t/W_{t-1}$. Note that if $m=4$, so $L_X'$ is of type $B_3$ and $(\operatorname {Isom}(W_t/W_{t-1}))'=D_4$, it may be that $L_X'$ acts irreducibly on