Inhibitory normalization in biological brains — where inhibitory interneurons regulate the activity of excitatory neurons — is well-studied as a mechanism for stabilizing sensory representations. But does it help learning?
Eyono et al. (arXiv:2603.17676) test this in artificial neural networks with separate excitatory and inhibitory populations. Normalizing only during inference (the forward pass) does not improve learning. Normalizing the error signals that drive learning (the backward pass) does — significantly.
The distinction is precise. During inference, normalization keeps activations in a useful range, preventing saturation and improving signal-to-noise. This is helpful but not special — any form of rescaling does something similar. During learning, normalization of error signals ensures that the credit assignment problem — which connections to strengthen, which to weaken — is solved relative to the local activity level, not in absolute terms. A neuron that fires rarely should update strongly when it does fire. A neuron that fires constantly should update weakly. Normalizing the error signal achieves this automatically.
The biological prediction is specific: if inhibition-mediated normalization improves learning in real brains, it must normalize the signals that drive synaptic plasticity, not just the signals that drive neural firing. The learning signals and the activity signals must both pass through the same normalization circuit. This constrains the wiring — the same inhibitory interneurons that normalize sensory responses must also access and normalize the error-related activity that modifies synapses.
Normalization of what you see is cosmetic. Normalization of what you learn from is structural.