Take the 2-minute tour ×
  1. Say I only have the mean vector and the covariance matrix of some multivaraite distribution X, where all single-variable marinals are normal (note: this is not generally a multinormal distribution). Is there some maximum distance, measured perhaps by some (multivariate) divergence like KL or JS Divergence, between X and Y, where Y is the multinormal defined by the mean vector and covraince matrix?

  2. Say I now only have a mean vector and the covariance matrix of a multivariate Bernoulli distribution. Is there a simple way to express the joint distribution probabilities of some multivariate Bernoulli that has such a mean vector and covariance matrix?

share|improve this question

1 Answer 1

up vote 2 down vote accepted

Regarding the second part of the question: say ${\bf x} = \{x_1, x_2 ... x_d\}$ is a $d-$dimensional Bernoulli variable. The joint probability function, in general, takes $2^d$ values, though the condition $\sum p({\bf x})=1$ leaves us with $2^d-1$ degrees of freedom. To represent it in some compact and handy way, the following representation is possible (I devised it some years ago for some problem I had at hands; it surely is well known).

I assume that our Bernoulli takes values in $\{-1,1\}$ instead of the more usual $\{0, 1\}$. Then, any joint probability of a set of Bernoullis can be written as (example for $\{ x_1,x_2,x_3 \}$) :

$$P(x_1,x_2,x_3)=\frac{1}{2^3}\left( a_{123} x_1 x_2 x_3 + a_{12} x_1 x_2 + a_{23} x_2 x_3 + a_{13} x_1 x_3 + a_{1} x_1 + a_{2} x_2 + a_{3} x_3 + a_{\emptyset} \right)$$

where $a_{i...k}=E[x_i ... x_k]$ (for example $a_{12}=E[x_1 x_2]$, etc) and $a_{\emptyset}=1$.

The convenience of this representation is twofold: first, the coefficients do not change if we add or suppress (marginalize) components: if we have the above formula and want to compute $P(x_1,x_2)$ or $P(x_1,x_2,x_3,x_4)$, we only must add or suppress terms, (and change the normalization factor). Second, independent components (recall that independent is the same as uncorrelated here) are easily spotted: for example, if $x_2$ is independent from $x_1$, then we'll have: $a_{12}=a_{1} a_2$. Further, $cov(x_1,x_2)=a_{12} - a_1 a_2$

The inconvenience of this representation is that, the condition $|a_{i...k}|\le1$ is necessary but not sufficient to be sure that it corresponds to a valid probability function. To check it, we must recover the values of the probability function. But the relation between this ${\bf p} =(p_{\emptyset},p_1,p_2 ... p_{12} ... p_{123})$ and the coefficients ${\bf a} =(a_{\emptyset},a_1,a_2 ... a_{12} ... a_{123})$ is straightforward: ${\bf a} = {\bf M p} $ with $m_{i,j}=\pm 1$ (the sign depends on the parity of common elements in the sets corresponding to that matrix row and column), and ${\bf M^{-1}}= {\bf M}^t / 2^d$ (I leave some details out, I doubt they will be missed).

So, if we are given the first and second moments of ${\bf x}$, we automatically have the first and second order coefficients. The rest of the coefficients are arbitrary, except for the restriction that they must lead to a valid probability function ($0 \le p_{i..k} \le 1$).

Regarding the first question, the idea is interesting, but I doubt there is some "maximum" distance. A mere suggestion: given that we are restricting to a variable with fixed first and second moments, to compute its deviation from the corresponding multivatiate gaussian I'd try with the entropy (substracted for the entropy for the corresponding gaussian, which should be bigger).

Added: To get explicitly ${\bf M}$: Note that in my notation for the joint probability function ${\bf p} =(p_{\emptyset},p_1,p_2 ... p_{12} ... p_{123})$ the subindexes denote the components that take positive value; hence, for example $p_{1}=P(x_1=1,x_2=-1,x_3=-1)$, $p_{23}=P(x_1=-1,x_2=1,x_3=1)$, etc.

Say we want to compute $a_{12}$.

$$a_{12}=E(x_1 x_2) = p_{123} (1)(1) + p_{12} (1)(1) + + p_{23} (-1)(1) + \cdots + p_1 (1)(-1) + p_{\emptyset}(-1)(-1)$$


$$a_i = \sum_j (-1)^{\#(S_i \setminus S_j)} p_j$$

where the indexes $i,j$ run over the $2^n$ subsets, and $\#(S_i \setminus S_j)$ is the cardinality of the difference set operation. That term, then, gives the elements of the matrix ${\bf M}$

Update: The matrix ${\bf M}$ is a type of Hadamard matrix, specifically (if I'm not mistaken) a Walsh matrix.

share|improve this answer
Thanks, leonbloy. However, I have two issues: [1] I need the Bernoulli values to be from {0,1}, since my mean vector and covariance matrix are based on this coding. [2] I actually need a closed form formula, that takes the mean vector and the covariance matrix and returns the joint probability, indexed say by k, where the binary coding of k of length d corresponding to (x1, x2,...xd). Your scheme is really an algorithm, whereas I need an expression (to insert as an element in a formula of mine). Note: if I assume all high moments are zero, is that sufficient to assure validity...? –  Omri Jul 19 '11 at 7:36
@Omri: [1] the conversion from {0,1} to {-1,1} is rather trivial, $x = 2z-1$, from this you get the transformed means and covariance –  leonbloy Jul 19 '11 at 11:47
@Omri: [2] I need a closed form... that returns the joint probability But: for the multivariate gaussian mean and covariance determine the joint probability; with multivariate Bernoulli, they dont: any valid multivariate probability with that mean and covariance, and taking the values in $\{0,1\}^n$ is a multivariate Bernoulli, there is not a "canonical" one, AFAIK. –  leonbloy Jul 19 '11 at 11:56
yes, of course, there is no canonical form, but I just wanted a closed form for p=(...). I guess I'm also not sure about how to construct M, where you write "the sign depends on the parity of common elements in the sets corresponding to that matrix row and column"...? –  Omri Jul 21 '11 at 0:10
@Omri: I added the explanation. I'm afraid you won't find it very "closed". The problem here is that, given moments and covariance there is no unique (or even priviledged) $p$. You can try some arbitrary values for the higher unknown $a$ (the value 0 only seems reasonable if the media is zero), and check if the result $p$ is valid. Perhaps this helps ideas.repec.org/a/eee/jmvana/v32y1990i2p256-268.html (I've not read it, all this is an old derivation of mine). –  leonbloy Jul 21 '11 at 2:39

Your Answer


