This research paper describes a simplistic architecture named as AANN: Absolute Artificial Neural Network, which can be used to create highly interpretable representations of the input data. These representations are generated by penalizing the learning of the network in such a way that those learned representations correspond to the respective labels present in the labelled dataset used for supervised training; thereby, simultaneously giving the network the ability to classify the input data. The network can be used in the reverse direction to generate data that closely resembles the input by feeding in representation vectors as required. This research paper also explores the use of mathematical abs (absolute valued) functions as activation functions which constitutes the core part of this neural network architecture. Finally the results obtained on the MNIST dataset by using this technique are presented and discussed in brief. In the field of philosophy, there has been a principle known as 'Ockham's Razor' which, in a simplified relevant language states that "Among the available multiple solutions to the same problem, the simplest one is the best one". For instance, if there are multiple polynomial functions that fit a given data distribution, the lowest degree one would be preferred BID12 . The technique AANN is driven by this principle. In spite of being elementary in its construction, an AANN is able to classify inputs in the forward direction while being able to generate them back in the reverse direction. It can be visualized to be doing classification in the forward direction whereas performing a regression task in the backward direction.A standalone GAN (Generative Adversarial Network) described in BID2 is able to create representations of the input data by using a novel technique of generating a distribution that contains the original data points as well as data points generated by the Generator part of the network; the distribution is then used by the Discriminator part of the network to classify the data points as genuine or generated. The representations generated by a GAN, although being very effective in creating undistinguishable data points, are however not interpretable and also highly entangled BID1 BID8 . Using an InfoGAN, the problem of entanglement is solved by training in such a way that the network maximises mutual information within small clusters of related latent representations BID1 . Auto-encoder is another technique that uses the concept of encoder-decoder architecture for creating low dimensional representations of the originally very high dimensional input data points. A VAE: Variational Auto-Encoder tries to make the learned representations sparse by using the KL-divergence cost as a regularizer on the final cost of an autoencoder BID5 . Various attempts at combining the two techniques of GAN and VAE have also been made in the unsupervised as well as semi-supervised learning directions BID8 BID7 . However, these techniques kept getting more and more complicated and somewhere in synthesizing these techniques, it is felt that the 'striving for simplicity' principle has been neglected.The Absolute Artificial Neural Network exploits all possible information available in the labelled training datasets to structure the learned representations of the input data. Structurally, an AANN is very similar to a feed forward Neural Network with the distinction that AANN uses the abs function as the activation function of the neurons. Due to this, all the activations produced, including the hidden layer activations, contain positive real number values. Thus, the network runs on the assumption that the input data as well as the label information comes from a positive data distribution. This doesn't create an issue for the computer vision based tasks. However, for those situations, where this is not possible, the feature values in the input dataset can be easily moved 1 into the positive region of the multi-dimensional input data space. The AANN transforms the n-dimensional input data into a space whose number of dimensions are equal to the number of labels used in the training dataset. For instance, presume that, the task is to classify images of cats and dogs and there is a labelled dataset present for achieving this classification. So, the learned representations will contain two dimensions corresponing to each label: cat and dog. The input images are transformed into 2-dimensional vectors by the AANN in such a way that the vectors are as close as possible to their ideal axes. This is achieved by constructing the cost function in a manner that it maximises the cosine value of the angle formed by the vector with its ideal axis. As a result, the representation space generated by this AANN can be visualized as shown in the FIG0 The AANN is constructed by using a 'Bidirectional Neuron' FIG1 ) as the building block for the hidden layers of a preliminary feed forward neural network. This bidirectional neuron uses the abs (mathematical absolute valued) function as the activation function. The computation performed by the neuron is similar in the forward and the backward directions. In the forward direction, the computation is given by: DISPLAYFORM0 Whereas, in the backward direction, the neuron computes: DISPLAYFORM1 The weights of the hidden layers of the AANN in forward direction learn to compute a function for transforming the input data into the representation vectors. While in the reverse direction, the weights constitute a function for constructing data points that closely resemble the data points belonging to the input dataset from the representation vectors. It is highly intriguing, and at the same time enigmatic, that the same set of weights constitute two entirely distinct functions. This research paper put forth an elementary but potent neural network architecture, named as AANN, that has the ability to learn in the forward as well as the backward direction. It also proposed the Abs function as a viable activation function for a neural network architecture. Due to lack of hardware resources, the experimentation had to be limited to the preliminary MNIST dataset, but it is firmly believed that the technique will perform equally well upon tackling other robust datasets, because of the theoretical evidence shown in the performed experiments.The AANN presently encodes the information in real number valued ranges across the the dedicated label axes in the the representation space. Certain regularization functions can be synthesized in order to stretch these ranges so that more information can be incorporated in them. The number of dimensions of the learned representations can be manually controlled by setting certain number of dedicated axes to a single label and by modifiying the forward cost function in such a way that the representation vectors lie inside the space generated by the coordinate axes dedicated to the ideal label. An in depth mathematical study of the Abs activation function could reveal the underlying behaviour of AANN. This forms the future scope for research.This technique also opens up new research opportunities for considering the AANN architectural modifications to certain network architectures like BID10 for semi-supervised learning. Moreover, it would be interesting to note the implications of applying the corresponding modifications to more advanced architectures such as Conv-nets BID6 and Recurrent Nets with LSTM cells BID3 .
Tied weights auto-encoder with abs function as activation function, learns to do classification in the forward direction and regression in the backward direction due to specially defined cost function. The paper proposes using the absolute value activation function in an autoencoder architecture with an additional supervised learning term in the objective function This paper introduces a reversible network with absolute value used as the activation function