Improved Auto-Encoding using Deterministic Projected Belief Networks
Paul M Baggenstoss
In this paper, we exploit the unique properties of a deterministic projected
belief network (D-PBN) to take full advantage of trainable compound activation
functions (TCAs). A D-PBN is a type of auto-encoder that operates by "backing
up" through a feed-forward neural network. TCAs are activation functions with
complex monotonic-increasing shapes that change the distribution of the data so
that the linear transformation that follows is more effective. Because a D-PBN
operates by "backing up", the TCAs are inverted in the reconstruction process,
restoring the original distribution of the data, thus taking advantage of a
given TCA in both analysis and reconstruction. In this paper, we show that a
D-PBN auto-encoder with TCAs can significantly out-perform standard
auto-encoders including variational auto-encoders.