Gradient descent updates weights. The loss gradient with respect to each parameter determines how that parameter changes. The optimization landscape is defined in weight space — a point is a set of parameters, a direction is a weight perturbation, and the geometry (loss surface curvature, saddle points, basins) is the geometry of weight space.
The paper shows that weight updates are more naturally understood as activation shifts. When a weight matrix W changes by δW, the activations at the next layer change by δW · x, where x is the input to that layer. The weight update δW is a matrix; the activation shift δW · x is a vector in the same space as the activations. The update that matters for the network's function is the activation shift, not the weight change.
This reframing is not merely notational. Different weight updates can produce identical activation shifts (if δW₁ · x = δW₂ · x for all relevant inputs). The effective dimensionality of the optimization is therefore lower in activation space than in weight space. Weight space has d_in × d_out dimensions per layer. Activation space has only d_out dimensions. The optimization is moving through a much lower-dimensional space than the parameterization suggests.
The practical consequence: optimization phenomena that seem puzzling in weight space become natural in activation space. The apparent over-parameterization of neural networks (far more parameters than data points) coexists with efficient optimization because the effective dimensionality is set by activations, not weights. The “flat directions” in the loss landscape that allow generalization correspond to weight directions that produce no activation shift — they are flat because they are irrelevant to the network's function.
The structural point: the native space of neural network optimization is not the space in which parameters live but the space in which representations live. The parameters are coordinates; the activations are the geometry. Optimizing in weight space is like navigating by latitude and longitude — correct but indirect. The activation view navigates by the terrain itself.