The Many Faces of Information Geometry

Information geometry Ama16AJLS17Ama21 aims at unravelling the geometric structures of families of probability distributions and at studying their uses in information sciences. Information sciences is an umbrella term regrouping statistics, information theory, signal processing, machine learning and AI, etc. Information geometry was born independently from econometrician H. Hotelling (1930) and statistician C. R. Rao (1945) from the mathematical curiosity of considering a parametric family of probability distributions, called the statistical model, as a Riemannian manifold equipped with the Fisher metric tensor Nie20. Information geometry tackles problems by using the concepts of differential geometry (like curvature) with tensor calculus. In his pioneer work, Rao considered the Riemannian geodesic distance and geodesic balls on the manifold to study classification and hypothesis testing problems in statistics.

Let $(\mathcal{X},\mathcal{F},\mu )$ denote a probability space Kee10 (with sample space $\mathcal{X}$, $\sigma$-algebra $\mathcal{F}$, and finite positive measure $\mu$, usually chosen as the Lebesgue mesure $\mu _L$ or the counting measure $\mu _c$), and consider a parametric family $\mathcal{P}=\{P_\theta \ :\ \theta \in \Theta \}$ of probability distributions, all dominated by $\mu$. Let $p_\theta (x)\coloneq \frac{dP_\theta (x)}{d\mu }$ denote the Radon-Nikodym derivative, the probability density function of random variable $X\sim p_\theta$. By definition, the Fisher Riemannian metric $g_F$ expressed in the $\theta$-coordinate system is the Fisher information matrix (FIM) of the random variable $X$: $[g_F]_\theta \coloneq I_X(\theta )$ with

$$\begin{equation*} I_X(\theta )\coloneq E_{p_\theta }\left[s_\theta (x) s_\theta (x)^\top \right], \end{equation*}$$

where $s_\theta (x)\coloneq \nabla _\theta \log p_\theta (x)$ is called the score function Kee10. The Fisher metric is also referred to as the Shahshahani metric in mathematical biology. Because the FIM is the covariance matrix of the score (since $E_{p_\theta }[s_\theta (x)]=0$), $I_X(\theta )$ is necessarily positive semidefinite, and positive-definite for regular statistical models Ama16. The FIM is covariant under reparameterization: for any smooth invertible mapping $\eta (\theta )$ with invertible Jacobian matrix $\left[\frac{\partial \theta _j}{\partial \eta _j}\right]_{ij}$, we have

$$\begin{equation*} I_\eta (\eta )=\left[\frac{\partial \theta _i}{\partial \eta _j}\right]^\top _{ij}\times I_\theta (\theta (\eta ))\times \left[\frac{\partial \theta _i}{\partial \eta _j}\right]_{ij}. \end{equation*}$$

The Riemannian Fisher length element induced by the Fisher metric

$$\begin{equation*} \mathrm{d}s_\theta =\sqrt {\mathrm{d}\theta ^\top \times [g_F] _\theta \times \mathrm{d}\theta } \end{equation*}$$

is invariant under any smooth invertible reparameterization: $\mathrm{d}s_\theta =\mathrm{d}s_\eta$ with $\mathrm{d}s^2_\eta =\mathrm{d}\eta ^\top \times [g_F]_\eta \times \mathrm{d}\eta$. Nowadays, the Riemannian manifold $(\mathcal{P},g_F)$ is commonly called the Fisher-Rao manifold Nie20, and its induced Riemannian geodesic length distance $\rho _g(\theta _1,\theta _2)$ is called the Fisher-Rao distance $\rho _\mathrm{Rao}(p_{\theta _1},p_{\theta _2})\coloneq \rho _{g_F}(\theta _1,\theta _2)$ with

$$\begin{equation*} \rho _{g_F}(\theta _1,\theta _2) \coloneq \int _0^1 \mathrm{d}s_{\gamma (t)}\, \mathrm{d}t, \end{equation*}$$

where $\gamma (t)$ denotes the Riemannian geodesic GN14 with the boundary conditions $\gamma (0)=\theta _1$ and $\gamma (1)=\theta _2$. Thus the Fisher-Rao distance used to evaluate the dissimilarities between probability distributions is invariant under reparameterization. For example, the Fisher-Rao distance remains the same whether the family of normal distributions are parameterized by $\lambda =(\mu ,\sigma )$, $\lambda '=(\mu ,\sigma ^2)$, or $\theta =(\frac{\mu }{\sigma ^2},-\frac{1}{2\sigma ^2})$:

$$\begin{equation*} \rho _\mathrm{Rao}(p_{\lambda _1},p_{\lambda _2})=\rho _\mathrm{Rao}(p_{\lambda _1'},p_{\lambda _2'})=\rho _\mathrm{Rao}(p_{\theta _1},p_{\theta _2}). \end{equation*}$$

This parameterization invariance property of statistical distances highlights the power of modeling geometrically statistical models. The Fisher-Rao manifolds may have different constant sectional curvatures $\kappa$: for example, the curvatures of the Fisher-Rao manifolds of univariate normal distributions, univariate zero-centered multivariate normal distributions, and categorical distributions are $\kappa =-\frac{1}{2}<0$, $\kappa =0$, and $\kappa =\frac{1}{4}>0$, respectively. Information geometry elucidates the role played by curvature in statistics. Since any Riemannian manifold of dimension $D$ can be isometrically embedded in a Euclidean space of dimension at most $D_E=\frac{1}{2}D(D+1)(3D+11)$ using Nash’s embedding theorem, we may visualize the Fisher-Rao manifold $(\mathcal{P},g_F)$ as a $D$-dimensional surface of $\mathbb{R}^{D_E}$. This extrinsic view of geometry is helpful to intuitively grasp the notion of tangent planes $T_{p_\theta }$ and tangent vectors $v\in T_{p_\theta }$ at any $p_\theta \in \mathcal{P}$ and allows one to visualize geodesics on surfaces. However, let us point out that differential geometry defines intrinsically these notions GN14.

Using the Fisher metric can be justified from several theoretical viewpoints Ama16:

•

First, the FIM occurs when locally approximating the Kullback-Leibler (KL) divergence Kee10. In statistics, estimating densities using the Maximum Likelihood Estimator (MLE) or the maximum entropy principle under moment constraints (MaxEnt) can be interpreted as KL divergence minimization problems Kee10 (to be detailed below). The KL divergence between densities $p_{\theta _1}$ and $p_{\theta _2}$ is defined by:$$\begin{equation*} D_\mathrm{KL}[p_{\theta _1}:p_{\theta _2}]\coloneq \int _{\mathcal{X}} p_{\theta _1}(x)\log \frac{p_{\theta _1}(x)}{p_{\theta _2}(x)}\mathrm{d}\mu (x). \end{equation*}$$

The delimiter “:” indicates that the divergence is oriented: $D_\mathrm{KL}[p_{\theta _1}:p_{\theta _2}]\not =D_\mathrm{KL}[p_{\theta _2}:p_{\theta _1}]$. The KL divergence can be expressed as$$\begin{equation*} D_\mathrm{KL}[p_{\theta _1}:p_{\theta _2}]=h^\times [p_{\theta _1}:p_{\theta _2}]-h[p_{\theta _1}], \end{equation*}$$

where $h^\times [p_{\theta _1}:p_{\theta _2}]$ denotes the cross-entropy $h^\times [p_{\theta _1}:p_{\theta _2}]\coloneq -\int p_{\theta _1}\log p_{\theta _2}\mathrm{d}\mu (x)$ and $h[p_{\theta _1}]\coloneq -\int p_{\theta _1}\log p_{\theta _1}\mathrm{d}\mu (x)$ is Shannon entropy. Hence, the KL divergence is also called relative entropy in information theory. The second-order Taylor approximation of the KL divergence yields$$\begin{equation*} D_\mathrm{KL}[p_{\theta }:p_{\theta +d\theta }]=\frac{1}{2}d\theta ^\top \times I_\theta (\theta )\times d\theta \approx \frac{1}{2}\mathrm{d}s_\theta ^2. \end{equation*}$$

More generally, the FIM is used in the local approximations of $f$-divergences Ama16:$$\begin{equation*} I_f[p_{\theta }:p_{\theta +d\theta }]=\frac{1}{2} f''(1)\, d\theta ^\top \times I_\theta (\theta )\times d\theta , \end{equation*}$$

where$$\begin{equation*} I_f[p_{\theta _1}:p_{\theta _2}] \coloneq \int _{\mathcal{X}} p_{\theta _1}(x)f\left(\frac{p_{\theta _2}(x)}{p_{\theta _1}(x)}\right)\mathrm{d}\mu (x) \end{equation*}$$

for a convex function $f(u)$ satisfying $f(1)=0$, and strictly convex at $1$. The KL divergence is an $f$-divergence obtained for $f(u)=-\log (u)$ with $f''(u)=1$. The $f$-divergences are said to be separable because they can be written as integrals of scalar divergences:$$\begin{equation*} I_f[p:q] = \int _{\mathcal{X}} i_f[p(x):q(x)] \mathrm{d}\mu (x) \end{equation*}$$

with $i_f[a:b]\coloneq a f(b/a)$. The $f$-divergences enjoy the following monotonicity property:$$\begin{equation*} I_f[p_Y:q_Y]\leq I_f[p_X:q_X], \end{equation*}$$

where $p_Y$ and $q_Y$ are the densities induced by a Markov kernel from measurable space $(\mathcal{X},\mathcal{F})$ to measurable space $(\mathcal{Y},\mathcal{G})$ Ama21. To give a concrete example, consider the $f$-divergences between two (normalized) histograms $p=(p_1,\ldots , p_{2n})$ and $q=(q_1,\ldots , q_{2n})$ with $2n$ bins representing multinomial probability laws. Then $I_f[p':q']\leq I_f[p:q]$, where $p'=(p_1',\ldots , p_{n}')$ and $q'=(q_1,\ldots , q_{n}')$ with $p_i'=p_{2(i-1)+1}+p_{2i}$ and $q_i'=q_{2(i-1)+1}+q_{2i}$ reduced histograms obtained by merging consecutive bins (a very special deterministic Markov kernel from measurable space $([2n],2^{[2n]})$ to measurable space $([n],2^{[n]})$ where $[s]\coloneq \{1,\ldots , s\}$). The only separable statistical divergences satisfying this monotonicity property are $f$-divergences when $n>2$ Ama16.

•

A sufficient statistic Kee10 $Y=T(X)$ for the parameter $\theta$ of a random variable $X\sim p_\theta$ is such that the conditional probability of $X$ given $Y=T(X)$ does not depend on $\theta$. That is, all statistical information concerning parameter $\theta$ is contained in $Y=T(X)$. For example, $T(X)=(T_1(X),T_2(X))$ with $T_1(X)=\sum _{i=1}^n X_i$ and $T_2(X)=\sum _{i=1}^n X_i^2$ is a sufficient statistic for the parameter $\theta =(\mu ,\sigma )$ of a random vector $(X_1,\ldots ,X_n)$ of $n$ independent and identically distributed (i.i.d.) random variables $X_1$, …, $X_n\sim p_\theta$ following a normal distribution $p_{\mu ,\sigma }$. In general, we have $I_{Y}(\theta )\leq I_X(\theta )$ with equality if and only if (iff) $Y$ is a sufficient statistic Kee10. Let us observe that for $n$ i.i.d. random variables $X_i\sim p_{\theta }$, we have $I_{X_1,\ldots , X_n}(\theta )=n I_X(\theta )$ with $X\sim p_\theta$. Sufficiency also characterizes the equality in the monotonicity inequality of $f$-divergences: $I_f[p_Y:q_Y]\leq I_f[p_X:q_X]$ with equality iff $Y=T(X)$ is a sufficient statistic.

•

Let $\hat{\theta }_n$ be an unbiased estimator of $\theta$ of an i.i.d. random vector $(X_1,\ldots ,X_n)\sim p_\theta$. Then the following Cramér-Rao lower bound (CRLB) on the variance of $\hat{\theta }_n$ holds:$$\begin{equation*} \mathrm{Var}[\hat{\theta }_n]=E\left[\left(\hat{\theta }_n-E[\hat{\theta }_n]\right)^2\right] \succeq \frac{1}{n}I_\theta ^{-1}(\theta ), \end{equation*}$$

where $A\succeq B$ iff matrix $A-B$ is positive semidefinite. The notation $\succeq$ indicates the comparison with respect to (w.r.t.) the Loewner partial ordering of positive semidefinite matrices. Thus the inverse of the FIM provides a lower bound on the accuracy of any unbiased estimator. An estimator is said to be Fisher efficient when its variance asymptotically matches the CRLB when $n\rightarrow \infty$.

The Fisher metric is the only invariant metric under Markovian morphisms of statistical models Ama16. However, let us point out that there are infinitely many counterparts of the FIM in quantum information geometry, and that other alternative Riemannian information metrics can be explored (e.g., the Wasserstein information metric Li21).

In statistics, two special types of statistical models, called the exponential families and the mixture families, are often handled:

•

An exponential family Kee10 is a set of parametric densities $\mathcal{E}\coloneq \{p_\theta (x)\mathrm{d}\mu \}$ such that$$\begin{equation*} p_\theta (x)\coloneq \exp \left(\sum _{i=1}^D t_i(x)\theta _i-F(\theta )\right), \end{equation*}$$

where the $(t_1(x),\ldots ,t_D(x))$ form the minimal sufficient statistic. Function $F(\theta )$ is used to normalize the densities:$$\begin{equation*} F(\theta )=\log \left(\int _\mathcal{X}\exp \left(\sum _{i=1}^D t_i(x)\theta _i\right) \mathrm{d}\mu (x)\right), \end{equation*}$$

and called the cumulant function (or log-partition in statistical physics). $F(\theta )$ is strictly convex for (full regular) exponential families Kee10. For example, the family of $d$-variate normal distributions is an exponential family of order $D=\frac{D(D+3)}{2}$ w.r.t. the Lebesgue measure $\mu _L$, and the family of Poisson distributions is a discrete exponential family of order $D=1$ w.r.t. the counting measure $\mu _c$. Exponential families have all finite-dimensional minimal sufficient statistics.

•

A mixture family Nie20 is a set of parametric densities $\mathcal{M}\coloneq \{m_\theta (x)\mathrm{d}\mu \}$ such that $m_\theta (x)=\sum _{i=1}^D \theta _i p_i(x)+ (1-\sum _{i=1}^D \theta _i) p_0(x)$, where functions $1$, $p_0(x)$, $p_1(x)$, …, $p_D(x)$ are linearly independent functions. Statistical mixtures such as Gaussian mixture models with prescribed component densities are examples of mixture families. Mixture families are closed under convex combinations. It can be shown that the negentropy of mixture densities is a strictly convex function Nie20: $F(\theta )=-h[m_\theta ]$.

For these two types of statistical models, the Fisher metric is a Hessian metric Shi07 since the FIM is the Hessian of some strictly convex potential function $F(\theta )$: $I(\theta )=\nabla ^2 F(\theta )$. This is easily checked for exponential families as the FIM can be written under mild regularity conditions Ama16 as $I_X(\theta )=-E_{p_\theta }[\nabla ^2\log p_\theta (x)]$.

In general, calculating in closed-form the Fisher-Rao distances may be difficult since it requires to solve the Riemannian geodesic equation with boundary conditions, and to integrate the length elements along geodesics. For example, although the Fisher-Rao distance between univariate normal distributions is available in closed-form, we do not have a closed-form formula for the Fisher-Rao distance between multivariate normals Nie20. Thus in practice, the Fisher-Rao between multivariate normals is numerically approximated. We shall now explain that the Fisher-Rao manifolds with Hessian metrics carry another beautiful geometric dual structure which is well suited for computation in applications: namely, these exponential/mixture families can be modeled as Hessian manifolds Shi07, and are commonly called dually flat spaces in information geometry Ama21.

In the second half of the 20th century, information geometry gained a momentum with the pathbreaking work of N. Chentsov. Chentsov shared statistician A. Wald’s viewpoint that all problems in statistics can be viewed as decision problems, and therefore investigated a theoretical framework for characterizing optimal decision rules in statistics using category theory and Markovian morphisms Čen82. A family of probability distributions should be invariant both under smooth one-to-one transformations of its parameter $\theta$ and under transformations of the corresponding random variables by sufficient statistics. This precisely defines the statistical invariance. Chentsov’s breakthrough consisted in separating the metric tensor $g$ (used to measure angles between vectors and lengths of vectors in a tangent plane) from its induced Levi-Civita connection ${}^{g}\nabla$ used by default on a Riemannian manifold for obtaining (locally) length minimizing geodesics. This novel insight allowed Chentsov to model statistical models as differentiable manifolds equipped with affine connections $\nabla$ more general than the Levi-Civita connection of Fisher-Rao manifolds. More precisely, Chentsov discovered the existence of a unique totally symmetric third-order tensor fulfilling the statistical invariance which is nowadays called the Amari-Chentsov tensor or skewness tensor, and used that tensor to build invariant affine connections. A. Kolmogorov called Chentsov’s field of research “geometrostatistics” in Russian (translated as “geometrical statistics” in the english monograph Čen82).

In short, an affine connection $\nabla$ GN14 defines the following:

•

A way to differentiate a vector field $X$ on a manifold (or more generally a tensor field $T$) by another vector field $Y$: namely, the covariant derivatives denoted by $\nabla _Y X$ (or $\nabla _Y T$).

•

The parallel transport $\prod _{c(t)}^\nabla v$ for a tangent vector $v\in T_{c(0)}$ along any smooth curve $c(t)$. An affine connection allows to parallel transport vectors of different tangent planes onto a common tangent plane in order to measure their subtended angles using the metric tensor in that common tangent plane.

•

$\nabla$-geodesics $\gamma$ defined as autoparallel curves: $\nabla _{\dot{\gamma }} \dot{\gamma }=0$. Geodesics $\gamma ^\nabla (t)$ are calculated by solving the second-order non-linear ordinary differential equation (ODE):$$\begin{equation*} \ddot{\theta }^{i}+\sum _{j, k=1}^{n} \Gamma _{j k}^{i} \dot{\theta }^{j} \dot{\theta }^{k}=0, \quad i\in \{1, \ldots , D\}, \end{equation*}$$

where $\theta =(\theta ^1,\ldots ,\theta ^D)$, $\dot{\theta }^{l}\coloneq \frac{d}{\mathrm{d}t}\theta ^l$, and $\Gamma _{j k}^{i}$ are the Christoffel symbols GN14 ($D^3$ smooth functions) defining the affine connection. In physics, geodesics represent free particle trajectories.

The curvature tensor $R^\nabla$ and the torsion tensor $T^\nabla$ of a manifold $(M,g,\nabla )$ are induced by the chosen connection $\nabla$ GN14. The fundamental theorem of Riemannian geometry GN14 states that there exists a unique torsion-free affine connection which preserves the metric, meaning that for any two vectors $v_1$ and $v_2$ of the tangent plane $T_p$, and a smooth curve $c(t)$ with $c(0)=p$, we have for any $t$: $g(v_1,v_2)=g\left(\prod _{c(t)}^\nabla v_1,\prod _{c(t)}^\nabla v_2\right)$. This unique torsion-free affine connection is called the Levi-Civita metric connection. Historically, affine connections were studied by É. Cartan in the 1920s, and used in the Einstein-Cartan theory of gravity. Chentsov considered regular exponential families in his monograph, and by considering their invariance, discovered the so-called exponential connection $\nabla ^e$.

Figure 1.

Fisher-Rao geometry vs. dual $\alpha$-geometry.

The field of information geometry was shaped by S.-i. Amari who dreamt of a mathematical theory of neuroscience. Amari pioneered the dualistic statistical structures of information geometry: that is, Amari showed that given any torsion-free affine connection $\nabla$, there exists a dual torsion-free affine connection $\nabla ^*$ such that the mid-connection $\frac{\nabla +\nabla ^*}{2}$ corresponds to the Levi-Civita connection. This duality ensures that the primal and dual parallel transports are metric compatible: $g(v_1,v_2)=g\left(\prod _{c(t)}^\nabla v_1,\prod _{c(t)}^{\nabla ^*} v_2\right)$. Notice that the lengths of dually parallel-transported vectors by $\nabla$ and $\nabla ^*$ may vary along $c(t)$ but their inner product is kept constant. For parametric statistical models $\mathcal{P}=\{p_\theta \}$, Amari reported the $\alpha$-geometry (for $\alpha \in \mathbb{R}$) which is a manifold equipped with the Fisher metric and a pair of dual connections $(\nabla ^{-\alpha },\nabla ^\alpha )$ coupled to the Fisher metric $g_F$: $\frac{\nabla ^{-\alpha }+\nabla ^\alpha }{2}=\nabla ^0={}^{g_F}\nabla$. Amari’s $\alpha$-connections $\nabla ^\alpha$ are defined by their Christoffel symbols $\Gamma ^\alpha _{ki,j}$:

$$\begin{equation*} \Gamma ^\alpha _{ki,j}(\theta )=E_{p_\theta }\left[\left(\partial _{k} \partial _{i} l_\theta (x)+\frac{1-\alpha }{2} \partial _{k} l_\theta (x) \partial _{i} l_\theta (x)\right) \partial _{j} l_\theta (x)\right], \end{equation*}$$

where $l_\theta (x)\coloneq \log p_\theta (x)$ is the log-likelihood function and $\partial _s\coloneq \frac{\partial }{\partial \theta _s}$. Chentsov’s exponential connection $\nabla ^e$ corresponds to Amari’s $\nabla ^1$ connection. The $e$-connection was also studied by Efron who defined geometrically the notion of statistical curvature to study the higher-order asymptotic theory of statistical estimators in a landmark paper Efr75 which has been recognized as one of the first successful applications of differential geometry to statistical inference. The dual connection of $\nabla ^e$ is $\nabla ^m\coloneq (\nabla ^e)^*=\nabla ^{-1}$, and called the mixture connection. This connection was proposed by P. Dawid, a discussant of Efron’s paper Efr75. Thus the Fisher-Rao geometry can be interpreted as the $0$-geometry enhanced with the Fisher-Rao geodesic distance (Figure 1). The Riemannian geodesics are ${}^g\nabla$-autoparallel and have the property to locally minimize the geodesic lengths GN14. In general, the $\alpha$-geometry is not associated with any statistical divergence when $\alpha \not =0$. But the $\alpha$-geometry may be recovered from the divergence geometry of invariant $f$-divergences on the probability simplex Egu83 (to be detailed below).

A connection $\nabla$ is said to be flat GN14Shi07 when it has zero torsion and when there exists a local coordinate system $\theta$ such that the Christoffel symbols $\Gamma _{ij}^k$ defining $\nabla$ expressed in that coordinate system vanish: $\Gamma _{ij}^k(\theta )=0$. The coordinate system $\theta$ is called a $\nabla$-affine coordinate system. In general the parallel transport of $v\in T_p$ to $T_q$ is curve dependent: $\prod _{c_1(t)}^\nabla v\not =\prod _{c_2(t)}^\nabla v$ for smooth curves $c_1$ and $c_2$ with endpoints $c_1(0)=c_2(0)=p$ and $c_1(1)=c_2(1)=q$. One can visualize locally the presence of curvature of a connection or not at a point $p$ by considering the parallel transport of a vector $v$ along a closed infinitesimal loop $l$ encircling $p$ (with $l(0)=l(1)$): if there is an angle deficiency between $v$ and $\prod _{l(1)}^\nabla v$, then the manifold has non-zero curvature at $p$ Nie20. However, the parallel transport is independent of the curves linking the point $p$ to the point $q$ for flat connections. It is a fundamental result of information geometry that if $(M,g,\nabla )$ is flat, then so is $(M,g,\nabla ^*)$ with $\nabla ^*=2{}^{g}\nabla -\nabla$. We get the so-called dually flat spaces of information geometry Ama16 $(M,g,\nabla ,\nabla ^*)$ which are special Hessian manifolds Shi07 admitting a single chart atlas. Notice that in a dually flat space, the Levi-Civita connection is usually not flat.

In information theory, a statistical divergence like the KL divergence is loosely speaking a potentially asymmetric dissimilarity measure between probability distributions which may fail the triangle inequality of metric distances. In information geometry, a divergence (historically called a contrast function Egu83) is a smooth dissimilarity measure $D(\theta :\theta ')$ between parameters $\theta$ and $\theta '$ that satisfies the following conditions:

(1)

$D(\theta :\theta ')\geq 0$ for all $\theta ,\theta '$ with equality iff $\theta =\theta '$.

(2)

${\partial _i}\left.D(\theta :\theta ')\right|_{\theta '=\theta }={\partial _j'}\left.D(\theta :\theta ')\right|_{\theta '=\theta }=0$ for all $i,j$, where $\partial _l\coloneq \frac{\partial }{\partial \theta _l}$ and $\partial _l'\coloneq \frac{\partial }{\partial \theta _l'}$.

(3)

$-\left[{\partial _i}{\partial _j'}\left.D(\theta :\theta ')\right|_{\theta '=\theta }\right]_{ij}$ is a positive-definite matrix.

The (parameter) divergence $D(\theta :\theta ')$ can also be interpreted as a function on the manifold defined by the single chart equipped with the $\theta$-coordinate system. Eguchi Egu83 reported a method to build a dualistic structure $(M,g,\nabla ,\nabla ^*)$ from any divergence $D(\cdot :\cdot )$ as follows:

$$\begin{align*} g_{ij}(\theta )&=-{\partial _i}{\partial _j'}\left.D(\theta :\theta ')\right|_{\theta '=\theta },\\ \Gamma _{ij,k}(\theta )&=-{\partial _i}{\partial _j}{\partial _k'}\left.D(\theta :\theta ')\right|_{\theta '=\theta },\\ \Gamma _{ij,k}^*(\theta ) &= -{\partial _k}{\partial _i'}{\partial _j'}\left.D(\theta :\theta ')\right|_{\theta '=\theta }. \end{align*}$$

It can be shown that the connections $\nabla$ and $\nabla ^*$ induced respectively by $\Gamma _{ij,k}$ and $\Gamma _{ij,k}^*$ are torsion-free and dual. This geometry is called the divergence geometry of $D$ Ama16. Let $D^*(\theta :\theta ')\coloneq D(\theta ':\theta )$ denote the dual or reverse divergence, and $(M,{}^Dg,{}^D\nabla ,{}^D\nabla ^*)$ the information-geometric space induced by $D$. Then we have ${}^{D^*}\nabla ={}^D\nabla ^*$ and ${}^{D^*}\nabla ^*={}^D\nabla$. Thus symmetric divergences $D(\theta :\theta ')=D(\theta ':\theta )$ yield self-dual connections coinciding with the Levi-Civita connection. Many different divergences may yield the same divergence geometry. The divergence geometry of $f$-divergences on the $D$-dimensional probability simplex corresponds to Amari’s $\alpha$-geometry for $\alpha =3+2\frac{f'''(1)}{f''(1)}$, and the divergence geometry of $D_\mathrm{Rao}(\theta _1:\theta _2)\coloneq \frac{1}{2}\rho _\mathrm{Rao}^2(p_{\theta _1},p_{\theta _2})$ yields the $0$-geometry.

Figure 2.

Dually flat space $(M,g,\nabla ,\nabla ^*)$ with $\nabla$-affine coordinate system $\theta$ and dual $\nabla ^*$-affine coordinate system $\eta$. Primal geodesics $\gamma ^\nabla$ and dual geodesics $\gamma ^{\nabla ^*}$ are linear when plotted in the $\theta$-coordinate system and $\eta$-coordinate system, respectively.

In a dually flat space, we can build a canonically Fenchel-Young (non-metric) divergence $A(\theta _1:\eta _2)$:

$$\begin{equation*} A(\theta _1:\eta _2)\coloneq F(\theta _1)+F^*(\eta _2)-\theta _1^\top \eta _2, \end{equation*}$$

where $F^*(\eta )$ denotes the convex conjugate obtained by the Legende-Fenchel transform:

$$\begin{equation*} F^*(\eta )=\sup _{\theta } \{\theta ^\top \eta -F(\theta )\}. \end{equation*}$$

Legende-Fenchel transform yields a dual coordinate system $\eta =\nabla F(\theta )$, and the Fenchel-Young inequality $F(\theta _1)+F^*(\eta _2)\geq \theta _1^\top \eta _2$ ensures that $A(\theta _1:\eta _2)\geq 0$ with equality iff $\eta _2=\nabla F(\theta _1)$. This divergence is shown to be equivalent to a Bregman divergence Ama16: $A(\theta _1:\eta _2)=B_F(\theta _1:\theta _2)$ with $\theta _2=\nabla F^*(\eta _2)$, where

$$\begin{equation*} B_F(\theta _1:\theta _2)\coloneq F(\theta _1)-F(\theta _2)-(\theta _1-\theta _2) ^\top \nabla F(\theta _2) \end{equation*}$$

is the Bregman divergence $B_F(\theta _1:\theta _2)$ between parameters $\theta _1$ and $\theta _2$ induced by a smooth strictly convex function $F(\theta )$ and $\nabla F(\theta )\coloneq \left[\frac{\partial F(\theta )}{\partial \theta _i}\right]_i^\top$ denotes the gradient of $F(\theta )$. Bregman divergences are widely used in machine learning and originated from mathematical programming. Reciprocally, given a Bregman divergence, we can build a dually flat space (the Bregman divergence geometry) with Hessian metric Shi07 $[g_{ij}(\theta )]_{ij}=\nabla ^2 F(\theta )$ (positive-definite because $F$ is strictly convex) from its divergence geometry. The dual affine connections are $\nabla$ and $\nabla ^*$ with Christoffel symbols $\Gamma _{ijk}(\theta )=0$ and ${\Gamma ^*}^{ijk}(\eta )=0$, respectively. It follows that in the $\theta$-coordinate system, the primal geodesics $\gamma ^\nabla$ are linear since the $\nabla$-geodesic ODE simplifies to $\ddot{\theta }^{i}=0$, and in the dual $\eta$-coordinate system, the dual geodesics are linear since the $\nabla ^*$-geodesic ODE simplifies to $\ddot{\eta }^{i}=0$ (Figure 2). The $\eta$-coordinate system is said to be $\nabla ^*$-affine. We have $[g_{ij}(\theta )]_{ij}=\nabla _\theta \nabla _\theta F(\theta )=\nabla _\theta \eta$ and the dual Riemannian metric $[g^{ij}(\eta )]_{ij}=\nabla _\eta \nabla _\eta F^*(\eta )=\nabla _\eta \theta$. It follows that $[g_{ij}(\theta )]_{ij}\, [g^{ij}(\eta )]_{ij}=I_{D\times D}$, the identity matrix. The Bregman divergence construction from a dually flat space is defined up to affine dual coordinate transformations $\theta '=A\theta +b$ and $\eta '=A^{-1}\eta +c$.

Amari’s $\pm 1$-geometry for the exponential and mixture families yields dually flat spaces. Their corresponding Bregman divergences yield the following statistical divergences:

•

For an exponential family Kee10 with density $p_\theta (x)=\exp (\sum _{i=1}^D t_i(x)\theta _i-F(\theta ))$, the Legendre-Fenchel conjugate function of the cumulant function $F(\theta )$ corresponds to Shannon negentropy, $F^*(\eta )=-h[P_\theta ]$, and the Bregman divergence $B_F(\theta _1:\theta _2)$ yields the reverse Kullback-Leibler divergence $D_\mathrm{KL}^*$ (or reverse relative entropy):$$\begin{equation*} B_F(\theta _1:\theta _2)=D_\mathrm{KL}^*[p_{\theta _1}:p_{\theta _2}]=D_\mathrm{KL}[p_{\theta _2}:p_{\theta _1}]. \end{equation*}$$

•

For a mixture family Ama16Nie20 $\mathcal{M}$ with density $m_\theta =\sum _{i=1}^D \theta _i p_i(x)+ (1-\sum _{i=1}^D \theta _i) p_0(x)$, the Bregman generator $F(\theta )=-h[m_\theta ]$ is strictly convex with $\Theta =\Delta _{D}^\circ$, the open $D$-dimensional probability simplex. The canonical Bregman divergence amounts to calculating the Kullback-Leibler divergence Nie20: $B_F(\theta _1:\theta _2)=D_\mathrm{KL}[m_{\theta _1}:m_{\theta _2}]$.

In a dually flat space $(M,g,\nabla ,\nabla ^*)$, a generalized Pythagorean theorem holds: Let $P$, $Q$, $R$ be three points. A primal geodesic $\gamma _{PQ}$ intersects a dual geodesic $\gamma ^*_{QR}$ orthogonally w.r.t. to the metric $g$ at point $Q$, written as $\gamma _{PQ}\perp _g \gamma ^*_{QR}$, iff

$$\begin{equation*} (\theta (P)-\theta (Q))^\top \times (\eta (R)-\eta (Q))=0. \end{equation*}$$

In that case, the following Pythagorean equality holds:

$$\begin{equation*} B_F(\theta (P):\theta (Q))+B_F(\theta (Q):\theta (R))=B_F(\theta (P):\theta (R)). \end{equation*}$$

When $F(\theta )=\frac{1}{2}\theta ^\top \theta$, we recover the usual Euclidean geometry (a self-dual flat space) with $\theta =\eta$ (since $\eta =\nabla F(\theta )=\theta$ and $\theta =\nabla F^*(\eta )=\eta$ with $F^*(\eta )=\frac{1}{2}\eta ^\top \eta$), and we have $B_F(\theta _1:\theta _2)=\frac{1}{2} \|\theta _1-\theta _2\|^2_2$, where $\|\cdot \|_2$ denotes the Euclidean norm. The canonical $\nabla$-divergence is $D_\nabla (P:Q)=B_F(\theta (P):\theta (Q))$ and the dual $\nabla ^*$-divergence is

$$\begin{align*} D_{\nabla ^*}(P:Q)&=B_{F^*}(\eta (P):\eta (Q))\\ &=B_F (\theta (Q):\theta (P))=D_{\nabla }(Q:P). \end{align*}$$

A $\theta$-flat is a submanifold $M'\subset M$ such that $\theta (M')\coloneq \{\theta (P)\ :\ P\in M'\}$ is an affine subspace of $\Theta$ ($M'$ is a $\nabla$-autoparallel submanifold). Similarly, an $\eta$-flat is a submanifold $M''$ such that $\eta (M'')\coloneq \{\eta (P)\ :\ P\in M''\}$ is an affine subspace of $H\coloneq \{\nabla F(\theta )\ :\ \theta \in \theta \}$. Define the $\nabla$-projection of a point $P\in M$ onto a submanifold $M'$ as

$$\begin{equation*} \mathrm{Proj}_{M'}^\nabla (P)\coloneq \{Q\in M'\ :\ \gamma ^\nabla _{PQ}\perp _g M'\}. \end{equation*}$$

Then the $\nabla$-projection of $P$ is guaranteed to be a unique point when $M'$ is an $\eta$-flat. Moreover, we have $\mathrm{Proj}_{M'}^\nabla (P)=\arg \min _{Q\in M'} D_\nabla (Q:P)$ in a dually flat space.

These information projections can be used in statistical inference as follows. Consider the probability space $(\mathcal{X},2^\mathcal{X},\mu _c)$ with finite discrete sample space $\mathcal{X}=\{1,\ldots , m\}$ and $\mu _c$ the counting measure. The categorical distributions form both an exponential family and a mixture family Ama16. A categorical probability mass function can be viewed as a point lying on the $(m-1)$-dimensional open standard simplex $\Delta _{m-1}$.

•

The Maximum Likelihood Estimator (MLE) of $n$ i.i.d. observations $x_1$, …, $x_n$ sampled from an exponential family density $p_\theta \in \mathcal{E}$ is $\hat{\eta }=\widehat{\nabla F(\theta )}=\frac{1}{n}\sum _{i=1}^n t(x_i)$. Since the MLE is equivariant Kee10, we have $\widehat{\nabla F(\theta )}=\nabla F(\hat{\theta })$, and it follows that $\hat{\theta }=(\nabla F)^*(\hat{\eta })$ since $(\nabla F)^{-1}=\nabla F^*$. The MLE can be interpreted as a divergence minimization problem: $\min _{\theta \in \Theta } D_\mathrm{KL}[p_e:p_\theta ]$, where $p_e(x)=\frac{1}{n}\sum _{i=1}^n \delta _{x_i}(x)$ is the empirical distribution with the $\delta _{x_i}$’s denoting the Dirac distributions $\delta _{x_i}(x)=1$ iff $x=x_i$, and $0$ otherwise. The MLE can be geometrically interpreted as an $m$-projection (i.e., with respect to $\nabla ^m$) of $p_e$ onto the $e$-flat exponential family: $p_{\hat{\theta }}=\mathrm{Proj}^{\nabla ^m}_\mathcal{E}(p_e)$. Thus the MLE $\hat{\theta }$ is unique since $\mathcal{E}$ is $\theta$-flat. This result holds more generally for estimations on curved exponential families $\mathcal{C}=\{p_{\theta (c)}\}\subset \mathcal{E}$ Ama16: for example, the family of normal distributions $p_{\mu ,1+\mu ^2}$ with $\theta (\mu )=(\mu ,1+\mu ^2)$ is a 1D curved exponential family. By viewing the MLE as a KL divergence minimization problem, we may consider other divergence-based estimators. A divergence $D$ yields a $D$-estimator by asking to solve the minimization problem $\min _{\theta \in \Theta } D[p_e:p_\theta ]$. The MLE is the $D_\mathrm{KL}$-estimator. Then we study the properties of various $D$-estimators. For example, the $D_\gamma$-estimator induced by the $\gamma$-divergence $D_\gamma$ for $\gamma >0$ is proven to be robust to noise contamination Ama16 but not the MLE which is based on the KL divergence. The $\gamma$-divergences tend to the KL divergence in the limit $\gamma \rightarrow 0$.

•

The Maximum Entropy (MaxEnt) principle of E. Jaynes Kee10 asks for the probability density $p(x)$ which maximizes the Shannon entropy under $D$ moment constraints $E_{p}[t_1(X)]=m_1$, …, $E_{p}[t_D(X)]=m_D$, i.e., $E_p[t(x)]=m$ with $t(x)=(t_1(x),\ldots , t_D(x))$ and $m=(m_1,\ldots ,m_D)$. It can be shown that the MaxEnt distribution $p^*$ is a density belonging to an exponential family $\mathcal{E}=\{p_\theta \}$ with sufficient statistics $t(x)$. Namely, we have $p^*=p_{\theta ^*_\mathrm{ME}}$ with $\eta ^*_\mathrm{ME}\coloneq \nabla F(\theta ^*_\mathrm{ME})=m$. The MaxEnt problem can be rewritten as the following minimization problem: $\min _{p\in \mathcal{M}} D_\mathrm{KL}[p:u]=\min _p D_\mathrm{KL}^*[u:p]$, where $u$ denotes the uniform distribution on the probability simplex $\Delta _{m-1}$, and $\mathcal{M}\coloneq \{p\in \Delta _{m-1}\ :\ E_p[t(x)]=m\}$ is an $m$-flat defined by the moment constraints. By introducing any other prior density $h(x)$, we can thus generalize MaxEnt by the following minimization problem: $\min _{p\in \mathcal{M}} D_\mathrm{KL}[p:h]$ under the moment constraint $E_p[t(x)]=m$. The MaxEnt solution $p^*$ belongs to an exponential family $\mathcal{E}\coloneq \{p_\theta (x)=\exp (\sum _i t_i(x)\theta _i-F(\theta ))h(x)\}$, and we have $p^*=p_{\theta ^*_\mathrm{ME}}$ such that $\eta ^*_\mathrm{ME}\coloneq \nabla F(\theta ^*_\mathrm{ME})=m$. We interpret the MaxEnt distribution $p_{\theta ^*_\mathrm{ME}}$ as the unique $e$-projection point (with respect to $\nabla ^e$) of $h$ onto $\mathcal{M}$ w.r.t. $\nabla ^e$: $p^*=\mathrm{Proj}_\mathcal{M}^{\nabla ^e}(h)$.

Wong Won18 recently generalized the Legendre-Fenchel transformation used in dually flat spaces, and obtained a new kind of Pythagorean theorem expressed w.r.t. Rényi divergences.

Figure 3.

Genesis of the dual structure of information geometry.

Finally, let us mention that instead of using the invariant $f$-divergences of information geometry, we can use the theory of optimal transport PC19 to measure the distance between any two probability measures. Optimal transport requires defining a ground distance between elements of the sample space to model the elementary cost of mass transportation, and measures the deviation between two probability measures by forward pushing one measure to another by a transportation plan. Although the optimal transport problems between discrete probability measures encountered in practice (i.e., finite weighted point sets) are computationally costly to solve (amount to solve linear programs), fast entropic-regularized methods PC19 and various heuristics like the sliced Wasserstein distances have contributed to its huge success in machine learning and computer vision. Optimal transport does not require the probability measures to have coinciding supports, and can even measure the distance between a discrete measure and a continuous measure. Many fruitful interactions between information geometry and optimal transport are investigated AKO18, and counterpart notions of the FIM and Bregman divergences have been proposed in the probability density space equipped with the $L_2$-Wasserstein metric Li21.

To summarize, the problem of geometrically modeling a family of probability distributions (the statistical model) is at the heart of information geometry. The Fisher-Rao geometry considers a Riemannian manifold equipped with the Fisher information metric, and uses the Riemannian geodesic length as a measure of dissimilarity between distributions: the Fisher-Rao distance. Amari’s dual $\pm \alpha$-geometry of information geometry has revealed the dualistic structure of affine connections coupled to the Fisher metric. This key dualistic structure is purely geometric and therefore can be used beyond the realm of statistics (for example, when studying optimization algorithms with convex barrier functions Ama16). Dually flat spaces are Hessian manifolds Shi07 with a single-chart atlas where the Legendre-Fenchel transformation plays an essential role to define dual coordinate systems and dual potential functions. Dually flat spaces generalize Euclidean geometry and enjoy a generalized Pythagorean theorem Ama16. Many pioneers have contributed to the now well-established classical dual structure of information geometry: Figure 3 displays historically the main actors who contributed to the genesis of the dual structure of information geometry with achieved milestones.

Recent advances in information geometry studies the geometry of deformed exponential families and their use in thermostatistics Nau11, the geometry of non-parametric models, the quantum information geometry, the Lie group thermodynamics, and the interactions of geometric mechanics with information geometry via symplectic and contact structures. Information geometry has found many applications beyond statistics. We refer to the textbook Ama16 for applications in signal processing, data science, and machine learning. To conclude with an application in machine learning, consider training a neural network $y=\mathrm{NN}_\theta (x)$ parameterized by weights $\theta$. The neural network is typically trained by using the method of gradient descent to minimize a loss function

$$\begin{equation*} L(\theta )\coloneq \frac{1}{n}\sum _{i=1}^n (y_i-\mathrm{NN}_\theta (x_i))^2 \end{equation*}$$

defined by a supervised training set of $n$ labeled pairs $\{(x_i,y_i)\}$, where $y_i$ denotes the label of $x_i$: initialize $\theta _0$ and iteratively update $\theta _{t+1}\coloneq \theta _t-\beta \nabla L(\theta _t)$, where $\beta$ denotes the step size. The ordinary gradient $\nabla _\theta L(\theta )$ depends on the chosen parameterization, i.e., $\nabla _\theta L(\theta )\not = \nabla _\eta L(\theta (\eta ))$ for a smooth invertible parameter transformation $\eta =\theta (\eta )$. A better parameter-invariant gradient has been proposed in information geometry for optimization on Riemannian manifolds $(M,g)$: the natural gradient Ama16 $\tilde{\nabla }_\theta L(\theta )\coloneq [g_{ij}(\theta )]_{ij}^{-1}\nabla _\theta \, L(\theta )$. The natural gradient ensures that $\tilde{\nabla }_\theta L(\theta )=\tilde{\nabla }_\eta L(\theta (\eta ))$. The natural gradient descent is used to train stochastic neural networks with parameter space modeled as a Fisher-Rao manifold, called a neuromanifold Ama16. Since 2018, an eponymous journal devoted to information geometry (INGE, https://www.springer.com/journal/41884) is published by Springer which reports the latest advances in the field.