Prime end rotation numbers of invariant separating contunua of annular homeomorphisms

Let $f$ be a homeomorphism of the closed annulus $A$ isotopic to the identity, and let $X\subset {\rm Int}A$ be an $f$-invariant continuum which separates $A$ into two domains, the upper domain $U_+$ and the lower domain $U_-$. Fixing a lift of $f$ to the universal cover of $A$, one defines the rotation set $\tilde \rho(X)$ of $X$ by means of the invariant probabilities on $X$, as well as the prime end rotation number $\check\rho_\pm$ of $U_\pm$. The purpose of this paper is to show that $\check\rho_\pm$ belongs to $\tilde\rho(X)$ for any separating invariant continuum $X$.


Introduction
Let f be a homeomorphism of the closed annulus A = S 1 × [−1, 1], isotopic to the identity, i. e. f preserves the orientation and each of the boundary components ∂ ± A = S 1 ×{±1}. Suppose there is an f -invariant partition of A; A = U − ∪X ∪U + , where U ± is a connected open set containing the boundary component ∂ ± A and X is a connected compact set. Let π :Ã = R × [−1, 1] → S 1 × [− 1,1] be the universal covering map and T :Ã →Ã a generator of the covering transformation group; T (ξ, η) = (ξ + 1, η). Denote by p :Ã → R the projection onto the first factor.
Fix once and for all a liftf :Ã →Ã of f . Then the function p •f − p is T -invariant and can be looked upon as a function on the annulus A. Define the rotation setρ(X) as the set of values µ(p •f − p), where µ ranges over the f -invariant probability measures supported on X. The rotation set is a compact interval (maybe one point) in R, which depends upon the choice of the liftf of f .
The first example of an invariant continuum X such that the frontiers of U ± satisfy Fr(U + ) = Fr(U − ) = X and that the rotation setρ(X) is not a singleton is constructed by G. D. Birkhoff in his 1932 year paper [B], and is refered to as a Birkhoff attractor. It turns out that the Birkhoff attractor is an indecomposable continuum ( [C, L2]). Furthermore it is shown by P. Le Calvez ([L1]) that for any rational number between the two prime end rotation numbers is realized by a correspoding periodic point off .
The purpose of this paper is to show the following.
This result is already known for X = Fr(U − ) = Fr(U + ) ( [BG]), and for any X if the homeomorphism f is area preserving (Lemma 5.4, [FL]).
It is shown in Theorem 2.2 of [F] that any rational number inρ(X) is realized by a periodic point if X consists of nonwandering points. Notice that then X, consisting of chain recurrent points, is chain transitive since it is connected, and thus satisfies the condition of Theorem 2.2. As a corollary we have Corollary 2. If X consists of nonwandering points and if p/q lies in the closed interval bounded byρ − andρ + , then there is a point x ∈ π −1 (X) such thatf q (x) = T p (x).
In what follows we also use the following notation. Leť Ψ ± :Ǔ ± → R × I ± be a lift of Ψ ± , and definep ± :Ǔ ± → R byp ± = p •Ψ ± . The projectionp ± is within a bounded error of p on π −1 (C) for a compact domain C of U ± . But they may be quite different on the whole π −1 (U ± ).

Proof
First of all let us state a deep and quite useful theorem of P. Le Calvez ([L3]) which plays a key role in the proof. A fixed point free and orientation preserving homeomorphism F of the plane R 2 is called a Brouwer homeomorphism. A proper oriented simple curve γ : ) is the right (resp. left) side complementary domain of γ, which is decided by the orientation of γ.
Theorem 2.1. Let F be a Brouwer homeomorphism commuting with the elements of a group Γ which acts on R 2 freely and properly discontinuously. Then there is a Γ-invariant oriented topological foliation of R 2 whose leaves are Brouwer lines of F .
The proof of Theorem 1 is by absurdity. Assume in way of contradiction thať ρ − < p/q < infρ(X). Considerngf q T −p instead off , it suffices to deduce a contradiction under the following assumption.
Since infρ(X) > 0, the mapf does not admit a fixed point in π −1 (X). The overall strategy of the proof is to modify the homeomorphism f away from X to a new one g without creating fixed points in A such that the restrictions ofg to the lifts of the both boundary circles π −1 (∂ ± A) are nontrivial rigid translations by the same translation number. Then by glueing the two boundary circles we obtain a torus T 2 and a homeomorphism on T 2 . Now we can apply Theorem 2.1 to the lift of the homeomorphism to the universal covering space. This yields a topological foliation on T 2 , which has long been well understood. The proof will be done by analyzing the foliation. We first prepare a lemma which is necessary for the desired modification. We do not presume Assumption 2.2 in the following.
Lemma 2.3. Assumef does not admit a fixed point in π −1 (X). Then the prime end rotation numberρ ± is nonzero.
Proof: Consider the mappingf − Id defined onÃ. Since it is T -invariant, it yields a mapping from A, still denoted by the same letter. Then since there is Clearly for any positively oriented essential simple closed curve γ in V , the degree of the mapf − Id : γ → R 2 \ {0} must be the same. If the curve γ is contained in U ± , then the degree can be studied by considering the mapf ± defined on the liftǓ ± of the prime end compactification U ± . If the prime end rotation numberρ ± is nonzero, the degree is clearly 0. Notice that our definition of the degree differs from the usual definition of the index.
To analyze the caseρ ± = 0, we need the following form of the Cartwright-Littlewood theorem [CL].
Let us complete the proof of Lemma 2.3. Theorem 2.4 enables us to compute the degree of the curve δ in U ± whenρ ± = 0. The degree is n if δ ⊂ U − and −n if δ ⊂ U + , where n is the number of the attractors. Since the degree must be the same in U − and U + , the conclusion follows. Now we haveρ − < 0 andρ + = 0 by Assumption 2.2 and Lemma 2.3. Let us start the modification of f . Lemma 2.5. Under Assumption 2.2, there exists a homeomorphism g of A such that (1) g = f in some neighbourhood of X, (2)g does not admit a fixed point inÃ, whereg is the lift of g such thatg =f on π −1 (X), (3)g is a negative rigid translation by the same translation number on π −1 (∂ ± A), and (4)p − •ǧ − −p − ≤ −c onÛ − for some positive number c.
can be established by a further obvious modification. Now to modify f in U + , we do the same thing as in U − . If the prime end rotation numberρ + is negative, then with an auxiliary modification we are done. If it is positive insert a time one map of the Reeb flow.
Consider the torus T 2 which is obtained from A by glueing the two boundary curves ∂ − A and ∂ + A. Then the condition (3) above shows that g induces a homeomorphism of T 2 , again denoted by g. The universal cover of T 2 is R 2 and A = R × [−1, 1] is a subset of R 2 . The liftg :Ã →Ã can be extended uniquely to a liftg : R 2 → R 2 of g : T 2 → T 2 . The covering transformation group Γ is isomorphic to Z 2 , generated by the horizontal translation T and the vertical translation by 2, denoted by S. Sinceg is a Brouwer homeomorphism which commutes with Γ, there is a Γ-invariant oriented foliation on R 2 whose leaves are Brouwer lines forg. This yields an oriented foliation F on the torus T 2 . The proof is divided into several cases according to the topological type of the foliation F . We are going to deduce a contradiction in each case. But before going into detail we need another lemma.
Lemma 2.6. For any C > 0 there is n > 0 such that p •g n − p ≥ C on X.
Proof: If not, there would be a point x n ∈ X for any n > 0 such that for some C > 0, and the averages of Dirac masses µ n = 1 n n−1 j=0 g j * δ xn would satisfy µ n (p •g − p) < C/n. Therefore an accumulation point µ of µ n would have the property that µ(p •g − p) ≤ 0, contradicting the assumption infρ(X) > 0.
Case 1. The foliation F does not admit a compact leaf. Then F is conjugate either to a linear foliation or to a Denjoy foliation, both of irrational slope. The lift F of F to the open annulus R 2 / T is conjugate to a foliation by vertical lines. The space of leaves ofF is homeomorphic to S 1 and there is a projection from R 2 / T to S 1 along the leaves of the foliation. This lifts to a projection q : R 2 → R. Now q restricted toÃ is within a bounded error of the first factor projection p :Ã → R that we have used for the definition of the rotation setρ(X). In fact both p and q are lifts of degree one maps from R 2 / T to S 1 and their difference is bounded on the preimageÃ = π −1 (A) of a compact subset A. Thus Lemma 2.6 shows that q •g n (x) → ∞ (n → ∞) for x ∈ π −1 (X). That is, the foliationF is oriented upward. But this shows that q•g(x) > q(x) even for a point x ∈ π −1 (∂ − A). On the other hand by condition (3) of Lemma 2.5,g is a negative translation on π −1 (∂ − A). A contradiction.
Case 2.1. The foliation F admits a compact leaf L of nonzero slope and does not admit a Reeb component. In this case the lifted foliationF is also conjugate to the vertical foliation and the argument of Case 1 applies.
Case 2.2. The foliation F admits a Reeb component R of nonzero slope. The Brouwer property of leaves implies that g(R) ⊂ Int(R) or g −1 (R) ⊂ Int(R). That is, a point of the boundary of R is wandering under g. Therefore ∂ − A, consisting of nonwandering points of g according to (3) of Lemma 2.5, cannot intersect the boundary of R, which is however impossible since the slope of R is nonzero.
Case 2.3. The foliation F admits a compact leaf of slope 0. Hereafter we only consider the dynamics and the foliation on the open annulus R 2 / T . Recall that A is a subset of R 2 / T , and the homeomorphism g on A is extended to the whole R 2 / T , again denoted by g, in such a way that g commutes with the vertical translation S, while the foliation is denoted byF as before.
Now the foliationF yields a partition P of the open annulus R 2 / T into compact leaves, interiors of Reeb components and foliated I-bundles. The set P is totally ordered by the height. The minimal element which intersects X cannot be a compact leaf by the Brouwer line property. Let R be the closure of the minimal element. Thus R is either a Reeb component or a foliated I-bundle such that Int(R) ∩ X = ∅ and ∂ − R ∩ X = ∅, where ∂ − R is the lower boundary curve of R.
Assume for a while that ∂ − R is oriented from the right to the left. Thus the homeomorphism g carries ∂ − R into the upper complement of ∂ − R.
On the other hand by condition (4) of Lemma 2.5, we havep • γ(t) → −∞ as t → ∞. In particular the curve γ is proper both inÃ and inǓ − pointing toward the opposite direction. By joining the point γ(0) to an appropriate point in π −1 (∂ − A), we obtain a simple curve δ in π −1 (U − ) starting at a point on π −1 (∂ − A) which extends γ.
Notice that there is a point of π −1 (X) on the left of a proper oriented curve δ inÃ, because the map p is bounded from below on δ and a high iterate of T −1 carries a point in π −1 (X) beyond that bound. (There might be a point of π −1 (X) on the right of δ however.) Let x be a point in π −1 (∂ − A) left to the initial point of δ. Then there is a simple path β : [0, ∞) → π −1 (U − ) such that β(0) = x, lim t→∞ β(t) ∈ π −1 (X), and β is disjoint from δ. The path β, extendable in π −1 (A) is also extendable inǓ − , the lift of the prime end compactification. (See e. g. Lemma 2.5 of [MN].) This implies that β defines a simple path inǓ − joining x to a prime end in π −1 (∂ ∞ U − ) without intersecting δ, which is impossible since π −1 (∂ ∞ U − ) is contained in the right side of the proper path δ inǓ − sincep − δ(t) → −∞, while x is on the left side. A contradiction.
But by condition (4) of Lemma 2.5, γ cannot intersectg −n (γ) for any large n. A contradiction.
Finally the case where ∂ − R is oriented from the left to the right can be dealt with similarly by reversing the time. This completes the proof of Theorem 1.