Designing neural networks for continuous-time stochastic processes is challenging, especially when observations are made irregularly. In this article, we analyze neural networks from a frame theoretic perspective to identify the sufficient conditions that enable smoothly recoverable representations of signals in L^2(R). Moreover, we show that, under certain assumptions, these properties hold even when signals are irregularly observed. As we converge to the family of (convolutional) neural networks that satisfy these conditions, we show that we can optimize our convolution filters while constraining them so that they effectively compute a Discrete Wavelet Transform. Such a neural network can efficiently divide the time-axis of a signal into orthogonal sub-spaces of different temporal scale and localization. We evaluate the resulting neural network on an assortment of synthetic and real-world tasks: parsimonious auto-encoding, video classification, and financial forecasting. The predominant assumption made in deep learning for time series analysis is that observations are made regularly, with the same duration of time separating each successive timestamps BID10 BID14 BID27 BID20 BID29 BID3 . However, this assumption is often inappropriate, as many real-world time series are observed irregularly and are, occasionally, event-driven (e.g., financial data, social networks, internet-of-things).One common approach in working with irregularly observed time series is to interpolate the observations to realign them to a regular time-grid. However , interpolation schemes may result in spurious statistical artifacts, as shown in BID17 BID4 . Fortunately , procedures for working with irregularly observed time series in their unaltered form have been devised, notably in the field of Gaussian-processes and kernel-learning BID17 BID4 and more recently in deep learning BID24 .In this article , we investigate the underlying representation of time series data as it is processed by a neural network. Our objective is to identify a class of neural networks that provably guarantee information preservation for certain irregularly observed signals. In doing so, we must analyze neural networks from a frame theoretic perspective, which has enabled a clear understanding of the impact discrete sampling has on representations of continuous-time signals BID7 BID5 BID6 BID13 BID15 BID22 .Although frame theory has historically been studied in the linear setting, recent work by BID26 has related frames with non-linear operators in Banach space, to what can be interpreted as non-linear frames. Here, we extend this generalization of frames to characterize entire families of neural networks. In doing so, we can show that the composition of certain non-linear neural layers (i.e., convolutions and fully-connected layers) form non-linear frames in L 2 (R), while others do not (i.e., recurrent layers).Moreover, frame theory can be used to analyze randomly-observed time series. In particular, when observations are made according to a family of self-exciting point processes known as Hawkes processes BID12 . We prove that such processes, under certain assumptions of stability, almost surely yield non-linear frames on a class of band-limited functions. That is to say , that despite having discrete and irregular observations, the signal of interest can still be smoothly recovered.As we obtain a family of convolutional neural networks that constitute non-linear frames, we show that under certain conditions, such networks can efficiently divide the time-axis of a time series into orthogonal sub-spaces of different temporal scale and localization. Namely, we optimize the weights of our convolution filters while constraining them so that they effectively compute a Discrete Wavelet Transform BID23 . Our numerical experiments on synthetic data highlight this unique capacity that allows neural networks to learn sparse representations of signals in L 2 (R), and how such a property is particularly powerful when training parsimoniously parameterized auto-encoders. Such auto-encoders learn optimal ways of compressing certain classes of input signals.Finally, we show that the ability of these networks to divide time series into a set sub-spaces, corresponding to different temporal scales and localization, can be composed with existing predictive frameworks to improve both accuracy and efficiency. This is demonstrated on real-world video classification and financial forecasting tasks. DISPLAYFORM0 We introduce the article with a theoretical analysis of the sufficient conditions on neural networks that enable smoothly recoverable representations of signals in L 2 (R) and prove that, under certain assumptions, this property holds true in the irregularly observed setting. In this article, we analyze neural networks from a frame theoretic perspective. In doing so, we come to the conclusion that by considering time series as an irregularly observed continuous-time stochastic processes, we are better able to devise robust and efficient convolutional neural networks. By leveraging recent contributions to frame theory, we prove properties about non-linear frames that allow us to make guarantees over an entire class of convolutional neural networks. Particularly regarding their capacity to produce discrete representations of continuous time signals that are both injective and bi-Lipschitz. Moreover, we show that, under certain conditions, these properties almost certainly hold, even when the signal is irregularly observed in an event-driven manner. Finally, we show that bounded-output recurrent neural networks do not satisfy the sufficient conditions to yield non-linear frames.This article is not limited to the theoretical statements it makes. In particular, we show that we can build a convolutional neural network that effectively computes a Discrete Wavelet Transform. The network's filters are dynamically learned while being constrained to produce outputs that preserve both orthogonality and the properties associated with non-linear frames. Our numerical experiments on real-world prediction tasks further demonstrate the benefits of such neural networks. Notably, their ability to produce compact representations that allow for efficient learning on latent continuous-time stochastic processes.
Neural architectures providing representations of irregularly observed signals that provably enable signal reconstruction. Proves that convolutional neural networks with Leaky ReLU activation function are nonlinear frames, with similar results for non-uniformly sampled time-series This article considers neural networks over time-series and show that the first convolutional filters can be chosen to represent a discrete wavelet transform.