Unsupervised End-to-End Training with a Self-Defined Bio-Inspired Target
Dongshu Liu Jérémie Laydevant Adrien Pontlevy Damien Querlioz Julie Grollier
Current unsupervised learning methods depend on end-to-end training via deep
learning techniques such as self-supervised learning, with high computational
requirements, or employ layer-by-layer training using bio-inspired approaches
like Hebbian learning, using local learning rules incompatible with supervised
learning. Both approaches are problematic for edge AI hardware that relies on
sparse computational resources and would strongly benefit from alternating
between unsupervised and supervised learning phases - thus leveraging widely
available unlabeled data from the environment as well as labeled training
datasets. To solve this challenge, in this work, we introduce a 'self-defined
target' that uses Winner-Take-All (WTA) selectivity at the network's final
layer, complemented by regularization through biologically inspired homeostasis
mechanism. This approach, framework-agnostic and compatible with both global
(Backpropagation) and local (Equilibrium propagation) learning rules, achieves
a 97.6% test accuracy on the MNIST dataset. Furthermore, we demonstrate that
incorporating a hidden layer enhances classification accuracy and the quality
of learned features across all training methods, showcasing the advantages of
end-to-end unsupervised training. Extending to semi-supervised learning, our
method dynamically adjusts the target according to data availability, reaching
a 96.6% accuracy with just 600 labeled MNIST samples. This result highlights
our 'unsupervised target' strategy's efficacy and flexibility in scenarios
ranging from abundant to no labeled data availability.