Sparse and Structured Hopfield Networks
Saul Santos Vlad Niculae Daniel McNamee Andre F. T. Martins
Modern Hopfield networks have enjoyed recent interest due to their connection
to attention in transformers. Our paper provides a unified framework for sparse
Hopfield networks by establishing a link with Fenchel-Young losses. The result
is a new family of Hopfield-Fenchel-Young energies whose update rules are
end-to-end differentiable sparse transformations. We reveal a connection
between loss margins, sparsity, and exact memory retrieval. We further extend
this framework to structured Hopfield networks via the SparseMAP
transformation, which can retrieve pattern associations instead of a single
pattern. Experiments on multiple instance learning and text rationalization
demonstrate the usefulness of our approach.