Deep neural networks use activation functions — ReLU, GELU, sigmoid, tanh, swish — to introduce nonlinearity between layers. Different layers may use different activations. The choice is empirical: practitioners try several and pick what works. The zoo of activation functions has grown steadily, each new function offering marginal improvements on specific benchmarks. The implicit assumption is that the specific nonlinearity matters.
The paper (arXiv:2603.12276, March 2026) replaces all activation functions with a single physics-inspired geometric kernel and shows that the network maintains universal approximation while improving training stability. One function replaces the entire zoo.
The kernel is derived from geometric principles — specifically, from the structure of kernel functions used in physics for spatial interpolation. It has a single mathematical form with no free parameters beyond what the network already has (weights and biases). The form is chosen for its geometric properties: smoothness, bounded derivatives, and natural scaling behavior. These properties prevent the pathological gradient dynamics (vanishing, exploding, dead neurons) that plague specific activation function choices.
The result does not improve state-of-the-art accuracy on benchmarks. It matches it. The contribution is simplification: one function does the work of many, with fewer failure modes and no tuning required. The zoo was unnecessary. The specific choice of nonlinearity that practitioners spend time optimizing contributes less to network performance than the architectural properties of the nonlinearity — smoothness, boundedness, appropriate curvature. Any function with the right geometric properties works. The zoo exists because practitioners search in function space when they should search in property space.
The structural lesson: when many different choices produce similar results, the choice is not the relevant variable — the shared property of the choices is. The activation function zoo is a search over instances when the search should be over properties. One function that embodies the necessary properties directly makes the search unnecessary.