On the Neural Tangent Kernel of Equilibrium Models
Zhili Feng J. Zico Kolter
This work studies the neural tangent kernel (NTK) of the deep equilibrium
(DEQ) model, a practical ``infinite-depth'' architecture which directly
computes the infinite-depth limit of a weight-tied network via root-finding.
Even though the NTK of a fully-connected neural network can be stochastic if
its width and depth both tend to infinity simultaneously, we show that
contrarily a DEQ model still enjoys a deterministic NTK despite its width and
depth going to infinity at the same time under mild conditions. Moreover, this
deterministic NTK can be found efficiently via root-finding.