Contraction Properties of the Global Workspace Primitive
Michaela Ennis Leo Kozachkov Jean-Jacques Slotine
To push forward the important emerging research field surrounding multi-area
recurrent neural networks (RNNs), we expand theoretically and empirically on
the provably stable RNNs of RNNs introduced by Kozachkov et al. in "RNNs of
RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural
Networks". We prove relaxed stability conditions for salient special cases of
this architecture, most notably for a global workspace modular structure. We
then demonstrate empirical success for Global Workspace Sparse Combo Nets with
a small number of trainable parameters, not only through strong overall test
performance but also greater resilience to removal of individual subnetworks.
These empirical results for the global workspace inter-area topology are
contingent on stability preservation, highlighting the relevance of our
theoretical work for enabling modular RNN success. Further, by exploring
sparsity in the connectivity structure between different subnetwork modules
more broadly, we improve the state of the art performance for stable RNNs on
benchmark sequence processing tasks, thus underscoring the general utility of
specialized graph structures for multi-area RNNs.