While deep neural networks have achieved groundbreaking prediction results in many tasks, there is a class of data where existing architectures are not optimal -- sequences of probability distributions. Performing forward prediction on sequences of distributions has many important applications. However, there are two main challenges in designing a network model for this task. First, neural networks are unable to encode distributions compactly as each node encodes just a real value. A recent work of Distribution Regression Network (DRN) solved this problem with a novel network that encodes an entire distribution in a single node, resulting in improved accuracies while using much fewer parameters than neural networks. However, despite its compact distribution representation, DRN does not address the second challenge, which is the need to model time dependencies in a sequence of distributions. In this paper, we propose our Recurrent Distribution Regression Network (RDRN) which adopts a recurrent architecture for DRN. The combination of compact distribution representation and shared weights architecture across time steps makes RDRN suitable for modeling the time dependencies in a distribution sequence. Compared to neural networks and DRN, RDRN achieves the best prediction performance while keeping the network compact. Deep neural networks have achieved state-of-the-art results in many tasks by designing the network architecture according to the data type. For instance, the convolutional neural network (CNN) uses local filters to capture the features in an image and max pooling to reduce the image representation size. By using a series of convolution and max pooling layers, CNN extracts the semantic meaning of the image. The recurrent architecture of recurrent neural networks (RNN) when unrolled, presents a shared weight structure which is designed to model time dependencies in a data sequence. However, among the major network architectures, the multilayer perceptron, convolutional neural network and recurrent neural network, there is no architecture suitable for representing sequences of probability distributions. Specifically, we address the task of forward prediction on distribution sequences.There are two main challenges in designing a network for sequences of probability distributions. First, conventional neural networks are unable to represent distributions compactly. Since each node encodes only a real value, a distribution has to be decomposed to smaller parts that are represented by separate nodes. When the distribution has been decomposed into separate nodes, the notion of distribution is no longer captured explicitly. Similarly, for image data, the fully-connected multilayer perceptron (MLP), unlike convolutional neural networks, fails to capture the notion of an image. A recently proposed network, Distribution Regression Network (DRN) BID8 , has solved this problem. DRN uses a novel representation of encoding an entire distribution in a single node, allowing DRN to use more compact models while achieving superior performance for distribution regression. It has been shown that DRN can achieve better accuracies with 500 times fewer parameters compared to MLP. However, despite the strengths of DRN, it is a feedforward network and hence it does not address a second problem, which is the need to model time dependencies in a distribution sequence.We address these two challenges and propose a recurrent extension of DRN, named the Recurrent Distribution Regression Network (RDRN). In the hidden states of RDRN, each node represents a distribution, thus containing much richer information while using fewer weights compared to the real-valued hidden states in RNN. This compact representation consequently results in better generalization performance. Compared to DRN, the shared weights in RDRN captures time dependencies better and results in better prediction performance. By having both compact distribution representations and modeling of time dependencies, RDRN is able to achieve superior prediction performance compared to the other methods. Neural network models work well by designing the architecture according to the data type. However, among the conventional neural network architectures, there is none that is designed for time-varying probability distributions. There are two key challenges in learning from distribution sequences. First, we require a suitable representation for probability distributions. Conventional neural networks, however, do not have suitable representations for distributions. As each node encodes only a real value, the distribution has to be split into smaller parts which are then represented by independent nodes. Hence, the neural network is agnostic to the distribution nature of the input data. A recently proposed Distribution Regression Network (DRN) addresses this issue. DRN has a novel network representation where each node encodes a distribution, showing improved accuracies compared to neural networks. However, a second challenge remains, which is to model the time dependencies in the distribution sequence. Both the recurrent neural network (RNN) and the Distribution Regression Network address only either one of the challenges. In this work, we propose our Recurrent Distribution Regression Network (RDRN) which extends DRN with a recurrent architecture. By having an explicit distribution representation in each node and shared weights across time steps, RDRN performs forward prediction on distribution sequences most effectively, achieving better prediction accuracies than RNN, DRN and other regression methods.
We propose an efficient recurrent network model for forward prediction on time-varying distributions. This paper proposes a method for creating neural nets that maps historical distributions onto distributions and applies the method to several distribution prediction tasks. Proposes a Reccurent Distribution Regression Network which uses a recurrent architecture upon a previous model Distribution Regression Network. This paper is on regressing over probability distributions by studying time varying distributions in a recurrent neural network setting