Equivalence of approximation by convolutional neural networks and fully-connected networks

Abstract:

Convolutional neural networks are the most widely used type of neural networks in applications. In mathematical analysis, however, mostly fully-connected networks are studied. In this paper, we establish a connection between both network architectures. Using this connection, we show that all upper and lower bounds concerning approximation rates of fully-connected neural networks for functions $f \in \mathcal {C}$—for an arbitrary function class $\mathcal {C}$—translate to essentially the same bounds concerning approximation rates of convolutional neural networks for functions $f \in \mathcal {C}^{\mathrm {equi}}$, with the class $\mathcal {C}^{\mathrm {equi}}$ consisting of all translation equivariant functions whose first coordinate belongs to $\mathcal {C}$. All presented results consider exclusively the case of convolutional neural networks without any pooling operation and with circular convolutions, i.e., not based on zero-padding.