Extracting relevant information, causally inferring and predicting the future states with high accuracy is a crucial task for modeling complex systems. The endeavor to address these tasks is made even more challenging when we have to deal with high-dimensional heterogeneous data streams. Such data streams often have higher-order inter-dependencies across spatial and temporal dimensions. We propose to perform a soft-clustering of the data and learn its dynamics to produce a compact dynamical model while still ensuring the original objectives of causal inference and accurate predictions. To efficiently and rigorously process the dynamics of soft-clustering, we advocate for an information theory inspired approach that incorporates stochastic calculus and seeks to determine a trade-off between the predictive accuracy and compactness of the mathematical representation. We cast the model construction as a maximization of the compression of the state variables such that the predictive ability and causal interdependence (relatedness) constraints between the original data streams and the compact model are closely bounded. We provide theoretical guarantees concerning the convergence of the proposed learning algorithm. To further test the proposed framework, we consider a high-dimensional Gaussian case study and describe an iterative scheme for updating the new model parameters. Using numerical experiments, we demonstrate the benefits on compression and prediction accuracy for a class of dynamical systems. Finally, we apply the proposed algorithm to the real-world dataset of multimodal sentiment intensity and show improvements in prediction with reduced dimensions. The use of machine learning for making inference and prediction from the real-world data has shown unprecedented growth. There exist a plethora of approaches for complex system (CS) modeling (e.g., multi-input multi-output state space identification (Stoica & Jansson, 2000) , expectation maximization (EM) BID17 , regularization BID6 , graphical models (Meinshausen & Buhlmann, 2006) , combined regularization and Bayesian learning BID11 BID10 BID3 , kernel-based regularization (Pillonetto & Chiuso, 2015) ). With the increase in the size of data, the complexity of the accurate models also increases, making inference and predictions slower. The major challenges of the upcoming era, hence, are likely to deal with the massive and diverse data sources, and still making quick decisions. Therefore, the compact modeling of time-varying complex systems 1 is a challenging task and appealing for more investigation. The real-world data has complex inter-dependencies across spatial and temporal dimensions. We aim to identify such dependencies and carefully construct a compact representation of the given CS model while still ensuring accurate predictions. We do so by performing a soft-clustering of such inter-dependencies to preserve only the relevant information. For the CS model in the form of a dynamical system, we additionally argue that similar to the data, the most relevant information also gets transformed at each hop in an alternate dynamical system. From a bird's-eye view, we track how the most relevant information propagates across the given dynamical system. We represent this propagation via an alternate dynamical system (compact model) and develop an unsupervised learning technique of such process.The most relevant work in this regard is information bottleneck (IB) principle (Tishby et al., 2000) . For fixed two random variables, it performs a soft-clustering to compress one variable while predicting another, given the joint probability distribution. The IB has been successfully applied to speech recognition (Hecht & Tishby, 2005) , document classification (Slonim & Tishby, 2000) , gene expression BID12 ) and deep learning (Tishby & Zaslavsky, 2015) , etc., and it has shown good performance. In contrast, we aim to learn a dynamics of the soft-clustering across the given dynamical system, and propose a general optimization framework to study the trade-offs between compactness and the resulting accuracies.The problem statement addressed in this work is: Given a dynamical system, we aim to develop a compact model by learning the dynamics of the soft-clustering in an unsupervised manner, or alternate dynamical process, through Information Bottleneck hierarchy (IBH). The main contributions of the present work are as follows: (i) By learning the dynamics of the soft-clustering, we propose an alternate compact dynamical system of the given process, with emphasis on the prediction accuracies. (ii) We formulate a novel optimization setup, compact perception problem, and characterize general solution to the information theoretic problem. (iii) We quantify how most relevant information about future gets transformed at each hop in the alternatively designed dynamical system.A brief mention of the mathematical notations is provided in the next part. In this paper, we have introduced a novel information-theoretic inspired approach to learn the compact dynamics of a time-varying complex system. The trade-off between the predictive accuracy and the compactness of the mathematical representation is formulated as a multi-hop compact perception optimization problem. A key ingredient to solve the aforementioned problem is to exploit variational calculus in order to derive the general solution expressions. Additionally, we have investigated the guaranteed convergence of the proposed iterative algorithm. Moreover, considering a specific class of distributions (Gaussian), we have provided closed-form expressions for the model parameters' update in our algorithm. Interestingly, the proposed compact perception shows improvements in prediction with reduced dimension on challenging real-world problems.The quantification of information flow across a dynamical system can have an enormous impact on understanding and improving the current state-of-the-art in neural networks as realized in (Tishby & Zaslavsky, 2015) . Moreover, modeling with dynamical systems is a standard approach, and by using the proposed framework, we can make a better compact representation of the system. The driving force of a dynamical system can enforce different behaviors of information flow, as realized in defining dynamical entropy by (Sinai, 1959) . Therefore, measuring the information flow can help in estimating/differentiating the actual driving component behind the observed activities. Such concepts are useful in predicting brain imagined tasks from observed electroencephalogram activities.The appendix is arranged as follows: In the Section A, we provide the proof of Theorem 1. In the Section B, we provide the iterative procedure (mentioned as Corollary 1) to minimize the functional in (4). Next, in Section C, we present the detailed proof of the Lemma 1, and finally, in Section D, a detailed proof of Theorem 2 is presented.A PROOF OF THEOREM 1Proof. For the sake of simplicity, a sketch of the proof is given for discrete variables. The Lagrangian associated with the minimization problem is the following DISPLAYFORM0 where α 1 (X k−1 ) and α 2 (B k ) are Lagrange multipliers for the normalization of the distributions p(B k |X k−1 ) and p(B k+1 |B k ), respectively. Taking the derivative of each term of the Lagrangian L with respect to p(B k |X k−1 ), we have DISPLAYFORM1 Setting the derivative of the Lagrangian equal to zero and arranging the terms we obtain the self consistent equation FORMULA7 . Note that all the constant terms in the derivative independent of B k will be captured by the Lagrange multiplier α 1 (X k−1 ). The derivative of the Lagrangian L with respect to p(B k+1 |B k ) involves only the two last terms from the functional F and the term that ensures the normalization condition. Then, we have DISPLAYFORM2 DISPLAYFORM3 Thus, the variational condition is written as follows DISPLAYFORM4 where α 2 (B k ) is the summation of the Lagrange multiplier α 2 (B k ) and the terms independent of B k+1 , and hence the equation FORMULA8 follows.
Compact perception of dynamical process Studies the problem of compactly representing the model of a complex dynamic system while preserving information by using an information bottleneck method. This paper studied the Gaussian linear dynamic and proposed an algorithm for computing the Information Bottleneck Hierarchy (IBH).