We achieve bias-variance decomposition for Boltzmann machines using an information geometric formulation. Our decomposition leads to an interesting phenomenon that the variance does not necessarily increase when more parameters are included in Boltzmann machines, while the bias always decreases. Our result gives a theoretical evidence of the generalization ability of deep learning architectures because it provides the possibility of increasing the representation power with avoiding the variance inflation. Understanding why the deep learning architectures can generalize well despite their high representation power with a large number of parameters is one of crucial problems in theoretical deep learning analysis, and there are a number of attempts to solve the problem with focusing on several aspects such as sharpness and robustness BID4 BID25 BID11 BID17 BID10 . However, the complete understanding of this phenomenon is not achieved yet due to the complex structure of deep learning architectures.To theoretically analyze the generalizability of the architectures, in this paper, we focus on Boltzmann machines BID0 and its generalization including higher-order Boltzmann machines BID20 BID14 , the fundamental probabilistic model of deep learning (see the book by Goodfellow et al. (2016, Chapter 20) for an excellent overview), and we firstly present bias-variance decomposition for Boltzmann machines. The key to achieve this analysis is to employ an information geometric formulation of a hierarchical probabilistic model, which was firstly explored by BID1 ; BID15 ; BID16 . In particular, the recent advances of the formulation by BID22 enables us to analytically obtain the Fisher information of parameters in Boltzmann machines, which is essential to give the lower bound of variances in bias-variance decomposition.We show an interesting phenomenon revealed by our bias-variance decomposition: The variance does not necessarily increase while the bias always monotonically decreases when we include more parameters in Boltzmann machines, which is caused by its hierarchical structure. Our result indicates the possibility of designing a deep learning architecture that can reduce both of bias and variance, leading to better generalization ability with keeping the representation power.The remainder of this paper is organized as follows: First we formulate the log-linear model of hierarchical probability distributions using an information geometric formulation in Section 2, which includes the traditional Boltzmann machines (Section 2.2) and arbitrary-order Boltzmann machines (Section 2.3). Then we present the main result of this paper, bias-variance decomposition for Boltzmann machines, in Section 3 and discuss its property. We empirically evaluate the tightness of our theoretical lower bound of the variance in Section 4. Finally, we conclude with summarizing the contribution of this paper in Section 5. In this paper, we have firstly achieved bias-variance decomposition of the KL divergence for Boltzmann machines using the information geometric formulation of hierarchical probability distributions. Our model is a generalization of the traditional Boltzmann machines, which can incorporate arbitrary order interactions of variables. Our bias-variance decomposition reveals the nonmonotonicity of the variance with respect to growth of parameter sets, which has been also reported elsewhere for non-linear models BID5 . This result indicates that it is possible to reduce both bias and variance when we include more higher-order parameters in the hierarchical deep learning architectures. To solve the open problem of the generalizability of the deep learning architectures, our finding can be fundamental for further theoretical development. Hidden layer 1 Hidden layer 2 1 2 3 4 {1} {2} {3} {4} {1,2} {1,2,3} {1,2,4} {1,3,4} {2,3,4} {1,3} {1,4} {2,3} {2,4} {1,2,3,4} {3,4} Ø {1} {2} {3} {4} {1,2} {1,2,3} {1,2,4} {1,3,4} {2,3,4} {1,3} {1,4} {2,3} {2,4} {1,2,3,4} {3,4} Ø Figure 3 : An example of a deep Boltzmann machine (left) with an input (visible) layer V = {1, 2} with two hidden layers H 1 = {3} and H 2 = {4}, and the corresponding domain set S V ∪H (right). In the right-hand side, the colored objects {1}, {2}, {3}, {4}, {1, 3}, {2, 3}, and {3, 4} denote the parameter set B, which correspond to nodes and edges of the DBM in the left-hand side.
We achieve bias-variance decomposition for Boltzmann machines using an information geometric formulation. The goal of this paper is to analyze the effectiveness and generalizability of deep learning by presenting a theoretical analysis of bias-variance decomposition for hierarchical models, specifically Boltzmann Machines   The paper arrives at the main conclusion that it is possible to reduce both the bias and the variance in a hierarchical model.