How deep must a residual network be to approximate a given function? The question sounds like it belongs to approximation theory — linear spaces, norms, rates. But deep networks approximate through composition, not linear combination. The right framework is geometric.
Cheng, Li, Lin, and Shen formalize this. A residual network with infinitesimal step size is a flow — a continuous trajectory in the space of diffeomorphisms. The family of allowed vector fields (determined by the architecture) defines which directions are accessible at each point. The minimal depth needed to approximate a target diffeomorphism is the minimal time to reach it along such a flow.
This minimal time is a geodesic distance on a sub-Finsler manifold. Sub-Finsler, not sub-Riemannian: the cost is measured by a norm on the accessible directions, and the norm need not come from an inner product. The local geometry is characterized by a variational principle tied to the vector field family — the Lie brackets of the accessible vector fields determine what can be reached and at what cost.
The result separates deep learning from linear approximation theory in a precise mathematical sense. Linear approximation measures distance from a function to a linear subspace. Deep approximation measures geodesic distance on a manifold of diffeomorphisms. The two frameworks have different geometry, different rates, different structural dependencies.
Depth is not a count of layers. It's a geometric distance — the length of the shortest path through the space of transformations the architecture can express.