Data with power-law eigenvalue decay — covariance eigenvalues falling as λ_k ~ k^{-α} — is ubiquitous: natural images, text embeddings, genomic data. Pass this data through a random feature layer (random linear projection followed by a nonlinear activation), and the output has its own spectral structure. Does the power law survive?
Yes. The output covariance inherits the exponent α exactly, modified only by a logarithmic correction that depends on the degree p of the activation function's dominant monomial term. The power law is preserved through the nonlinear transformation; only the sub-leading behavior changes.
The proof uses dyadic decomposition — splitting the spectrum into octaves — combined with Wick chaos expansions that decompose the nonlinearity into orthogonal polynomial components. Each monomial degree p contributes its own spectral decay, and the dominant contribution preserves the input exponent. Random matrix concentration handles the finite-width effects.
The matching upper and lower bounds are the key technical achievement. An upper bound alone would say “the output can't have a slower decay than the input.” The matching lower bound says “the output can't have a faster decay either.” The spectral structure is genuinely inherited, not just bounded.
This matters for understanding depth. If each layer preserves the spectral exponent, then deep networks process power-law data in the power-law regime regardless of depth. The spectral structure is a conserved quantity of the random feature transformation — the nonlinearity changes the representation but preserves this statistical fingerprint.