A generalization of the Lefschetz fixed point theorem and detection of chaos

Abstract:

We consider the problem of existence of fixed points of a continuous map $f:X\to X$ in (possibly) noninvariant subsets. A pair $(C,E)$ of subsets of $X$ induces a map $f^\dagger :C/E\to C/E$ given by $f^\dagger ([x])=[f(x)]$ if $x,f(x)\in C\setminus E$ and $f^\dagger ([x])=[E]$ elsewhere. The following generalization of the Lefschetz fixed point theorem is proved: If $X$ is metrizable, $C$ and $E$ are compact ANRs, and $f^\dagger$ is continuous, then $f$ has a fixed point in $\overline {C\setminus E}$ provided the Lefschetz number of $\widetilde H^\ast (f^\dagger )$ is nonzero. Actually, we prove an extension of that theorem to the case of a composition of maps. We apply it to a result on the existence of an invariant set of a homeomorphism such that the dynamics restricted to that set is chaotic.