Proceedings of the American Mathematical Society

Abstract:

For any $C$, $A_1$, $A_2$, $A_3 \in {\frak {so}}(2n+1)$, let $W_C(A_1, A_2, A_3)$ be the following subset of ${\mathbb R}^3$: \[ \{(\operatorname {tr} CO^TA_1O, \operatorname {tr} CO^TA_2O, \operatorname {tr} CO^TA_3O): O\in SO(2n+1)\}. \] We show that if $n\ge 2$, then $W_C(A_1, A_2, A_3)$ is always convex. When $n = 1$, it is an ellipsoid, probably degenerate. The convexity result is best possible in the sense that if we have $W_C(A_1, \dots , A_p)$ defined similarly, then there are examples which fail to be convex when $p \ge 4$ and $n\ge 1$.

The set is also symmetric about the origin for all $n\ge 1$, and contains the origin when $n \ge 2$. Equivalent statements of this result are given. The convexity result for ${\frak {so}}(2n+1)$ is similar to Au-Yeung and Tsing’s extension of Westwick’s convexity result for ${\frak u}(n)$.