On Feature Diversity in Energy-based Models
Firas Laakom Jenni Raitoharju Alexandros Iosifidis Moncef Gabbouj
Energy-based learning is a powerful learning paradigm that encapsulates
various discriminative and generative approaches. An energy-based model (EBM)
is typically formed of inner-model(s) that learn a combination of the different
features to generate an energy mapping for each input configuration. In this
paper, we focus on the diversity of the produced feature set. We extend the
probably approximately correct (PAC) theory of EBMs and analyze the effect of
redundancy reduction on the performance of EBMs. We derive generalization
bounds for various learning contexts, i.e., regression, classification, and
implicit regression, with different energy functions and we show that indeed
reducing redundancy of the feature set can consistently decrease the gap
between the true and empirical expectation of the energy and boosts the
performance of the model.