<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 $$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\Parents}{\mathrm{Parents}}
\newcommand{\NonDesc}{\mathrm{NonDesc}}
\newcommand{\I}{\mathcal{I}}
\newcommand{\dsep}{\text{d-sep}}
$$
Step1:  EECS 445
Step2:  Example
Step3:  Example
Step4:  Example
Step5:  Other Latent Variable Models
Many other models in machine learning involve latent variables
Step6:  Answer 1
Step7:  Answer 2
Step8:  Answer 3
Step9:  Answer 4
Step10:  Answer 5
Step11:  Answer
Step12:  Example
Step13:  Example
<ASSISTANT_TASK:>
Python Code:
from __future__ import division
# scientific
%matplotlib inline
from matplotlib import pyplot as plt;
import numpy as np;
import sklearn as skl;
import sklearn.datasets;
import sklearn.cluster;
# ipython
import IPython;
# python
import os;
#####################################################
# image processing
import PIL;
# trim and scale images
def trim(im, percent=100):
    print("trim:", percent);
    bg = PIL.Image.new(im.mode, im.size, im.getpixel((0,0)))
    diff = PIL.ImageChops.difference(im, bg)
    diff = PIL.ImageChops.add(diff, diff, 2.0, -100)
    bbox = diff.getbbox()
    if bbox:
        x = im.crop(bbox)
        return x.resize(((x.size[0]*percent)//100, (x.size[1]*percent)//100), PIL.Image.ANTIALIAS);
#####################################################
# daft (rendering PGMs)
import daft;
# set to FALSE to load PGMs from static images
RENDER_PGMS = False;
# decorator for pgm rendering
def pgm_render(pgm_func):
    def render_func(path, percent=100, render=None, *args, **kwargs):
        print("render_func:", percent);
        # render
        render = render if (render is not None) else RENDER_PGMS;
        
        if render:
            print("rendering");
            # render
            pgm = pgm_func(*args, **kwargs);
            pgm.render();
            pgm.figure.savefig(path, dpi=300);
            
            # trim
            img = trim(PIL.Image.open(path), percent);
            img.save(path, 'PNG');
        else:
            print("not rendering");
        
        # error
        if not os.path.isfile(path):
            raise "Error:  Graphical model image %s not found.  You may need to set RENDER_PGMS=True.";
        
        # display
        return IPython.display.Image(filename=path);#trim(PIL.Image.open(path), percent);
    
    return render_func;
######################################################
Explanation: $$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\Parents}{\mathrm{Parents}}
\newcommand{\NonDesc}{\mathrm{NonDesc}}
\newcommand{\I}{\mathcal{I}}
\newcommand{\dsep}{\text{d-sep}}
$$
End of explanation
X, y = skl.datasets.make_blobs(1000, cluster_std=[1.0, 2.5, 0.5], random_state=170)
plt.scatter(X[:,0], X[:,1])
Explanation: EECS 445:  Machine Learning
Lecture 16:  Latent Variables, d-Separation, Gaussian Mixture Models
Instructor:  Jacob Abernethy
Date:  November 9, 2016
References
[MLAPP] Murphy, Kevin. Machine Learning:  A Probabilistic Perspective.  2012.
[PRML] Bishop, Christopher.  Pattern Recognition and Machine Learning.  2006.
[Koller & Friedman 2009] Koller, Daphne and Nir Friedman.  Probabilistic Graphical Models.  2009.
Book Chapter: The language of directed acyclic
graphical models
This notes are really nice
Outline
Review of Exponential Families
MLE
Brief discussion of conjugate priors and MAP estimation
Probabilistic Graphical Models
Review of Conditional Indep. Assumptions
Intro to Hidden Markov Models
Latent Variable Models in general
d-separation in Bayesian Networks
Mixture Models
Gaussian Mixture Model
Relationship to Clustering
Exponential Family Distributions
DEF: $p(x | \theta)$ has exponential family form if:
$$
\begin{align}
p(x | \theta)
&= \frac{1}{Z(\theta)} h(x) \exp\left[ \eta(\theta)^T \phi(x) \right] \
&= h(x) \exp\left[ \eta(\theta)^T \phi(x) - A(\theta) \right]
\end{align}
$$
$Z(\theta)$ is the partition function for normalization
$A(\theta) = \log Z(\theta)$ is the log partition function
$\phi(x) \in \R^d$ is a vector of sufficient statistics
$\eta(\theta)$ maps $\theta$ to a set of natural parameters
$h(x)$ is a scaling constant, usually $h(x)=1$
Exponential Family:  MLE
To find the maximum, recall $\nabla_\theta A(\theta) = E_\theta[\phi(x)]$, so
\begin{align}
\nabla_\theta \log p(\D | \theta) & =
\nabla_\theta(\theta^T \phi(\D) - N A(\theta)) \
& = \phi(\D) - N E_\theta[\phi(X)] = 0
\end{align}
Which gives
$$E_\theta[\phi(X)] = \frac{\phi(\D)}{N} = \frac{1}{N} \sum_{k=1}^N \phi(x_k)$$
Obtaining the maximum likelihood is simply solving the calculus problem $\nabla_\theta A(\theta) = \frac{1}{N} \sum_{k=1}^N \phi(x_k)$ which is often easy
Bayes for Exponential Family
Exact Bayesian analysis is considerably simplified if the prior is conjugate to the likelihood.
- Simply, this means that prior $p(\theta)$ has the same form as the posterior $p(\theta|\mathcal{D})$.
This requires likelihood to have finite sufficient statistics
* Exponential family to the rescue!
Note: We will release some notes on cojugate priors + exponential families. It's hard to learn from slides and needs a bit more description. 
Likelihood for exponential family
Likelihood: 
$$ p(\mathcal{D}|\theta) \propto g(\theta)^N \exp[\eta(\theta)^T s_N]\
s_N = \sum_{i=1}^{N}\phi(x_i)$$
In terms of canonical parameters:
$$ p(\mathcal{D}|\eta) \propto \exp[N\eta^T \bar{s} -N A(\eta)] \
\bar s = \frac{1}{N}s_N $$
Conjugate prior for exponential family
The prior and posterior for an exponential family involve two parameters, $\tau$ and $\nu$, initially set to $\tau_0, \nu_0$
$$ p(\theta| \nu_0, \tau_0) \propto g(\theta)^{\nu_0} \exp[\eta(\theta)^T \tau_0] $$
Denote $ \tau_0 = \nu_0 \bar{\tau}_0$ to separate out the size of the prior pseudo-data, $\nu_0$ , from the mean of the sufficient statistics on this pseudo-data, $\tau_0$ . Hence,
$$ p(\theta| \nu_0, \bar \tau_0) \propto \exp[\nu_0\eta(\theta)^T \bar \tau_0  - \nu_0 A(\eta)] $$
Think of $\tau_0$ as a "guess" of the future sufficient statistics, and $\nu_0$ as the strength of this guess
Prior:  Example
$$
\begin{align}
p(\theta| \nu_0, \tau_0) 
&\propto (1-\theta)^{\nu_0} \exp[\tau_0\log(\frac{\theta}{1-\theta})] \
&= \theta^{\tau_0}(1-\theta)^{\nu_0 - \tau_0}
\end{align}
$$
Define $\alpha = \tau_0 +1 $ and $\beta = \nu_0 - \tau_0 +1$ to see that this is a beta distribution.
Posterior
Posterior: 
$$ p(\theta|\mathcal{D}) = p(\theta|\nu_N, \tau_N) = p(\theta| \nu_0 +N, \tau_0 +s_N) $$
Note that we obtain hyper-parameters by adding. Hence,
$$ \begin{align}
p(\eta|\mathcal{D})
&\propto \exp[\eta^T (\nu_0 \bar\tau_0 + N \bar s) - (\nu_0 + N) A(\eta) ] \
&= p(\eta|\nu_0 + N, \frac{\nu_0 \bar\tau_0 + N \bar s}{\nu_0 + N})
\end{align}$$
where $\bar s = \frac 1 N \sum_{i=1}^{N}\phi(x_i)$.
posterior hyper-parameters are a convex combination of the prior mean hyper-parameters and the average of the sufficient statistics.
Back to Graphical Models!
Recall: Bayesian Networks:  Definition
A Bayesian Network $\mathcal{G}$ is a directed acyclic graph whose nodes represent random variables $X_1, \dots, X_n$.
- Let $\Parents_\G(X_k)$ denote the parents of $X_k$ in $\G$
- Let $\NonDesc_\G(X_k)$ denote the variables in $\G$ who are not descendants of $X_k$.
Examples will come shortly...
Bayesian Networks:  Local Independencies
Every Bayesian Network $\G$ encodes a set $\I_\ell(\G)$ of local independence assumptions:
For each variable $X_k$, we have $(X_k \perp \NonDesc_\G(X_k) \mid \Parents_\G(X_k))$
Every node $X_k$ is conditionally independent of its nondescendants given its parents.
Example:  Naive Bayes
The graphical model for Naive Bayes is shown below:
- $\Parents_\G(X_k) = { C }$, $\NonDesc_\G(X_k) = { X_j }_{j\neq k} \cup { C }$
- Therefore $X_j \perp X_k \mid C$ for any $j \neq k$
<img src="../lecture15_exp_families_bayesian_networks/images/naive-bayes.png">
Factorization Theorem:  Statement
Theorem: (Koller & Friedman 3.1) If $\G$ is an I-map for $P$, then $P$ factorizes as follows:
$$
P(X_1, \dots, X_N) = \prod_{k=1}^N P(X_k \mid \Parents_\G(X_k))
$$
Example:  Fully Connected Graph
A fully connected graph makes no independence assumptions.
$$
P(A,B,C) = P(A) P(B|A) P(C|A,B)
$$
<img src="../lecture15_exp_families_bayesian_networks/images/fully-connected-b.png">
Importan PGM Example:  Markov Chain
State at time $t$ depends only on state at time $t-1$.
$$
P(X_0, X_1, \dots, X_N) = P(X_0) \prod_{t=1}^N P(X_t \mid X_{t-1})
$$
<img src="../lecture15_exp_families_bayesian_networks/images/markov-chain.png">
Compact Representation of PGM:  Plate Notation
We can represent (conditionally) iid variables using plate notation.
A box around a variable (or set of variables), with a $K$, means that we have access to $K$ iid samples of this variable.
<img src="../lecture15_exp_families_bayesian_networks/images/plate-example.png">
Unobserved Variables:  Hidden Markov Model
Noisy observations $X_k$ generated from hidden Markov chain $Y_k$. (More on this soon)
<img src="../lecture15_exp_families_bayesian_networks/images/hmm.png">
This Brings us to: Latent Variable Models
Uses material from [MLAPP] §10.1-10.4, §11.1-11.2
Latent Variable Models
In general, the goal of probabilistic modeling is to 
Use what we know to make inferences about what we don't know.
Graphical models provide a natural framework for this problem.
- Assume unobserved variables are correlated due to the influence of unobserved latent variables.
- Latent variables encode beliefs about the generative process.
In a graphical model, we will often shade in the observed variables to distinguish them from hidden variables.
Example:  Gaussian Mixture Models
This dataset is hard to explain with a single distribution.
- Underlying density is complicated overall...
- But it's clearly three Gaussians!
End of explanation
@pgm_render
def pgm_gmm():
    pgm = daft.PGM([4,4], origin=[-2,-1], node_unit=0.8, grid_unit=2.0);
    # nodes
    pgm.add_node(daft.Node("pi", r"$\pi$", 0, 1));
    pgm.add_node(daft.Node("z", r"$Z_j$", 0.7, 1));
    pgm.add_node(daft.Node("x", r"$X_j$", 1.3, 1, observed=True));
    pgm.add_node(daft.Node("mu", r"$\mu$", 0.7, 0.3));
    pgm.add_node(daft.Node("sigma", r"$\Sigma$", 1.3, 0.3));
    
    # edges
    pgm.add_edge("pi", "z", head_length=0.08);
    pgm.add_edge("z", "x", head_length=0.08);
    pgm.add_edge("mu", "x", head_length=0.08);
    pgm.add_edge("sigma", "x", head_length=0.08);
    
    pgm.add_plate(daft.Plate([0.4,0.8,1.3,0.5], label=r"$\qquad\qquad\qquad\;\; N$",
    shift=-0.1))
    
    return pgm;
%%capture
pgm_gmm("images/pgm/pgm-gmm.png")
Explanation: Example:  Mixture Models
Instead, introduce a latent cluster label $z_j \in [K]$ for each datapoint $x_j$,
$$
\begin{align}
z_j &\sim \mathrm{Cat}(\pi_1, \dots, \pi_K) 
    & \forall\, j=1,\dots,N \
x_j \mid z_j &\sim \mathcal{N}(\mu_{z_j}, \Sigma_{z_j})
    & \forall\, j=1,\dots,N \
\end{align}
$$
This allows us to explain a complicated density as a mixture of simpler densities:
$$
P(x | \mu, \Sigma) = \sum_{k=1}^K \pi_k \mathcal{N}(x | \mu_k, \Sigma_k)
$$
Example:  Mixture Models
End of explanation
@pgm_render
def pgm_hmm():
    pgm = daft.PGM([7, 7], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("Y1", r"$Y_1$", 1, 3.5))
    pgm.add_node(daft.Node("Y2", r"$Y_2$", 2, 3.5))
    pgm.add_node(daft.Node("Y3", r"$\dots$", 3, 3.5, plot_params={'ec':'none'}))
    pgm.add_node(daft.Node("Y4", r"$Y_N$", 4, 3.5))
    pgm.add_node(daft.Node("x1", r"$X_1$", 1, 2.5, observed=True))
    pgm.add_node(daft.Node("x2", r"$X_2$", 2, 2.5, observed=True))
    pgm.add_node(daft.Node("x3", r"$\dots$", 3, 2.5, plot_params={'ec':'none'}))
    pgm.add_node(daft.Node("x4", r"$X_N$", 4, 2.5, observed=True))
    # Add in the edges.
    pgm.add_edge("Y1", "Y2", head_length=0.08)
    pgm.add_edge("Y2", "Y3", head_length=0.08)
    pgm.add_edge("Y3", "Y4", head_length=0.08)
    pgm.add_edge("Y1", "x1", head_length=0.08)
    pgm.add_edge("Y2", "x2", head_length=0.08)
    pgm.add_edge("Y4", "x4", head_length=0.08)
    
    return pgm;
%%capture
pgm_hmm("images/pgm/hmm.png");
Explanation: Example:  Hidden Markov Models
Noisy observations $X_k$ generated from hidden Markov chain $Y_k$.
$$
P(\vec{X}, \vec{Y}) = P(Y_1) P(X_1 \mid Y_1) \prod_{k=2}^N \left(P(Y_k \mid Y_{k-1}) P(X_k \mid Y_k)\right)
$$
End of explanation
@pgm_render
def pgm_unsupervised():
    pgm = daft.PGM([6, 6], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("d1", r"$Z_1$", 2, 3.5))
    pgm.add_node(daft.Node("di", r"$Z_2$", 3, 3.5))
    pgm.add_node(daft.Node("dn", r"$Z_3$", 4, 3.5))
    pgm.add_node(daft.Node("f1", r"$X_1$", 1, 2.50, observed=True))
    pgm.add_node(daft.Node("fi-1", r"$X_2$", 2, 2.5, observed=True))
    pgm.add_node(daft.Node("fi", r"$X_3$", 3, 2.5, observed=True))
    pgm.add_node(daft.Node("fi+1", r"$X_4$", 4, 2.5, observed=True))
    pgm.add_node(daft.Node("fm", r"$X_N$", 5, 2.5, observed=True))
    # Add in the edges.
    pgm.add_edge("d1", "f1", head_length=0.08)
    pgm.add_edge("d1", "fi-1", head_length=0.08)
    pgm.add_edge("d1", "fi", head_length=0.08)
    pgm.add_edge("d1", "fi+1", head_length=0.08)
    pgm.add_edge("d1", "fm", head_length=0.08)
    pgm.add_edge("di", "f1", head_length=0.08)
    pgm.add_edge("di", "fi-1", head_length=0.08)
    pgm.add_edge("di", "fi", head_length=0.08)
    pgm.add_edge("di", "fi+1", head_length=0.08)
    pgm.add_edge("di", "fm", head_length=0.08)
    pgm.add_edge("dn", "f1", head_length=0.08)
    pgm.add_edge("dn", "fi-1", head_length=0.08)
    pgm.add_edge("dn", "fi", head_length=0.08)
    pgm.add_edge("dn", "fi+1", head_length=0.08)
    pgm.add_edge("dn", "fm", head_length=0.08)
    
    return pgm
%%capture
pgm_unsupervised("images/pgm/unsupervised.png");
Explanation: Example:  Unsupervised Learning
Latent variables are fundamental to unsupervised and deep learning.
- Serve as a bottleneck
- Compute a compressed representation of data
End of explanation
@pgm_render
def pgm_question1():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("c", r"$C$", 2, 3.5))
    pgm.add_node(daft.Node("a", r"$A$", 1.3, 2.5))
    pgm.add_node(daft.Node("b", r"$B$", 2.7, 2.5))
    # Add in the edges.
    pgm.add_edge("c", "a", head_length=0.08)
    pgm.add_edge("c", "b", head_length=0.08)
    return pgm;
%%capture
pgm_question1("images/pgm/question1.png")
Explanation: Other Latent Variable Models
Many other models in machine learning involve latent variables:
Neural Networks / Multilayer Perceptrons
Restricted Boltzmann Machines
Deep Belief Networks
Probabilistic PCA
Latent Variable Models:  Complexity
Latent variable models exhibit emergent complexity.
- Although each conditional distribution is simple,
- The joint distribution is capable of modeling complex interactions.
However, latent variables make learning difficult.
- Inference is challening in models with latent variables.
- They can introduce new dependencies between observed variables.
Break time
<img src="images/boxing_cat.gif"/>
Bayesian Networks
Part II:  Inference, Learning, and d-Separation
Uses material from [Koller & Friedman 2009] Chapter 3, [MLAPP] Chapter 10, and [PRML] §8.2.1
Bayesian Networks:  Terminology
Typically, our models will have
- Observed variables $X$
- Hidden variables $Z$
- Parameters $\theta$
Occasionally, we will distinguish between inference and learning.
Bayesian Networks:  Inference
Inference:  Estimate hidden variables $Z$ from observed variables $X$.
$$
P(Z | X,\theta) = \frac{P(X,Z | \theta)}{P(X|\theta)}
$$
Denominator $P(X|\theta)$ is sometimes called the probability of the evidence.
Occasionally we care only about a subset of the hidden variables, and marginalize out the rest.
Bayesian Networks:  Learning
Learning: Estimate parameters $\theta$ from observed data $X$.
$$
P(\theta \mid X) = \sum_{z \in Z} P(\theta, z \mid X) = \sum_{z \in Z} P(\theta \mid z, X) P(z \mid X)
$$
To Bayesians, parameters are hidden variables, so inference and learning are equivalent.
Bayesian Networks:  Probability Queries
In general, it is useful to compute $P(A|B)$ for arbitrary collections $A$ and $B$ of variables.
- Both inference and learning take this form.
To accomplish this, we must understand the independence structure of any given graphical model.
Review:  Local Independencies
Every Bayesian Network $\G$ encodes a set $\I_\ell(\G)$ of local independence assumptions:
For each variable $X_k$, we have $(X_k \perp \NonDesc_\G(X_k) \mid \Parents_\G(X_k))$
Every node $X_k$ is conditionally independent of its nondescendants given its parents.
For arbitrary sets of variables, when does $(A \perp B \mid C)$ hold?
Review:  I-Maps
If $P$ satisfies the independence assertions made by $\G$, we say that
- $\G$ is an I-Map for $P$
- or that $P$ satisfies $\G$.
Any distribution satisfying $\G$ shares common structure.
- We will exploit this structure in our algorithms
- This is what makes graphical models so powerful!
Review:  Factorization Theorem
Last time, we proved that for any $P$ satisfying $\G$,
$$
P(X_1, \dots, X_N) = \prod_{k=1}^N P(X_k \mid \Parents_\G(X_k))
$$
If we understand independence structure, we can factorize arbitrary conditional distributions:
$$
P(A_1, \dots, A_n \mid B_1, \dots, B_m) = \;?
$$
Question 1:  Is $(A \perp B)$?
End of explanation
@pgm_render
def pgm_question2():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("c", r"$C$", 2, 3.5, 
                           observed=True))
    pgm.add_node(daft.Node("a", r"$A$", 1.3, 2.5))
    pgm.add_node(daft.Node("b", r"$B$", 2.7, 2.5))
    # Add in the edges.
    pgm.add_edge("c", "a", head_length=0.08)
    pgm.add_edge("c", "b", head_length=0.08)
    
    return pgm
%%capture
pgm_question2("images/pgm/question2.png")
Explanation: Answer 1:  No!
No!  $A$ and $B$ are not marginally independent.
- Note $C$ is not shaded, so we don't observe it.
In general, 
$$
P(A,B) = \sum_{c \in C} P(A,B,c) = \sum_{c \in C} P(A|c)P(B|c)P(c) \neq P(A)P(B)
$$
Question 2:  Is $(A \perp B \mid C)$?
End of explanation
@pgm_render
def pgm_question3():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("c", r"$C$", 2, 3.5))
    pgm.add_node(daft.Node("a", r"$A$", 1.3, 2.5))
    pgm.add_node(daft.Node("b", r"$B$", 2.7, 2.5))
    # Add in the edges.
    pgm.add_edge("a", "c", head_length=0.08)
    pgm.add_edge("c", "b", head_length=0.08)
    
    return pgm
%%capture
pgm_question3("images/pgm/question3.png")
Explanation: Answer 2:  Yes!
Yes!  $(A \perp B | C)$ follows from the local independence properties of Bayesian networks.
Every variable is conditionally independent of its nondescendants given its parents.
Observing $C$ blocks the path of influence from $A$ to $B$. Or, using factorization theorem:
$$
\begin{align}
P(A,B|C) & = \frac{P(A,B,C)}{P(C)}  \
& = \frac{P(C)P(A|C)P(B|C)}{P(C)} \
& = P(A|C)P(B|C)
\end{align}
$$
Question 3:  Is $(A \perp B)$?
End of explanation
@pgm_render
def pgm_question4():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("c", r"$C$", 2, 3.5, observed=True))
    pgm.add_node(daft.Node("a", r"$A$", 1.3, 2.5))
    pgm.add_node(daft.Node("b", r"$B$", 2.7, 2.5))
    # Add in the edges.
    pgm.add_edge("a", "c", head_length=0.08)
    pgm.add_edge("c", "b", head_length=0.08)
    
    return pgm
%%capture
pgm_question4("images/pgm/question4.png")
Explanation: Answer 3:  No!
Again, $C$ is not given, so $A$ and $B$ are dependent.
Question 4:  Is $(A \perp B \mid C)$?
End of explanation
@pgm_render
def pgm_question5():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("c", r"$C$", 2, 3.5))
    pgm.add_node(daft.Node("a", r"$A$", 1.3, 2.5))
    pgm.add_node(daft.Node("b", r"$B$", 2.7, 2.5))
    # Add in the edges.
    pgm.add_edge("a", "c", head_length=0.08)
    pgm.add_edge("b", "c", head_length=0.08)
    
    return pgm
%%capture
pgm_question5("images/pgm/question5.png")
Explanation: Answer 4:  Yes!
Again, observing $C$ blocks influence from $A$ to $B$.
Every variable is conditionally independent of its nondescendants given its parents.
Question 5:  Is $(A \perp B)$?
End of explanation
@pgm_render
def pgm_question6():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("c", r"$C$", 2, 3.5, observed=True))
    pgm.add_node(daft.Node("a", r"$A$", 1.3, 2.5))
    pgm.add_node(daft.Node("b", r"$B$", 2.7, 2.5))
    # Add in the edges.
    pgm.add_edge("a", "c", head_length=0.08)
    pgm.add_edge("b", "c", head_length=0.08)
    
    return pgm
%%capture
pgm_question6("images/pgm/question6.png")
Explanation: Answer 5:  Yes!
Using the factorization rule,
$$
P(A,B,C) = P(A)P(B)P(C\mid A,B)
$$
Therefore, marginalizing out $C$,
$$
\begin{align}
P(A,B) & = \sum_{c \in C} P(A,B,c)  \
& = \sum_{c \in C} P(A)P(B) P(c \mid A,B) \
& = P(A)P(B) \sum_{c \in C} P(c \mid A,B) = P(A)P(B)
\end{align}
$$
Question 6:  Is $(A \perp B \mid C)$?
End of explanation
@pgm_render
def pgm_bfg_1():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("G", r"$G$", 2, 3.5))
    pgm.add_node(daft.Node("B", r"$B$", 1.3, 2.5))
    pgm.add_node(daft.Node("F", r"$F$", 2.7, 2.5))
    # Add in the edges.
    pgm.add_edge("B", "G", head_length=0.08)
    pgm.add_edge("F", "G", head_length=0.08)
    
    return pgm;
%%capture
pgm_bfg_1("images/pgm/bfg-1.png")
Explanation: Answer:  No!
$A$ can influence $B$ via $C$.
$$
P(A,B | C) = \frac{P(A,B,C)}{P(C)} = \frac{P(A)P(B)P(C|A,B)}{P(C)}
$$
This does not factorize in general to $P(A|C)P(B|C)$.
Example:  Battery, Fuel, and Gauge
Consider three binary random variables
- Battery $B$ is either charged $(B=1)$ or dead, $(B=0)$
- Fuel tank $F$ is either full $(F=1)$ or empty, $(F=0)$
- Fuel gauge $G$ either indicates full $(G=1)$ or empty, $(G=0)$
Assume $(B \perp F)$ with priors
- $P(B = 1) = 0.9$
- $P(F = 1) = 0.9$
Example:  Battery, Fuel, and Gauge
Given the state of the fuel tank and the battery, the fuel gauge reads full with probabilities: 
 - $p(G = 1 \mid B = 1, F = 1) = 0.8$
 - $p(G = 1 \mid B = 1, F = 0) = 0.2$
 - $p(G = 1 \mid B = 0, F = 1) = 0.2$
 - $p(G = 1 \mid B = 0, F = 0) = 0.1$
Example:  Battery, Fuel, and Gauge
Without any observations, the probability of an empty fuel tank is
$$
P(F=0) = 1 - P(F = 1) = 0.1
$$
End of explanation
@pgm_render
def pgm_bfg_2():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("G", r"$G$", 2, 3.5, offset=(0, 20), observed=True))
    pgm.add_node(daft.Node("B", r"$B$", 1.3, 2.5, offset=(0, 20)))
    pgm.add_node(daft.Node("F", r"$F$", 2.7, 2.5, offset=(0, 20)))
    # Add in the edges.
    pgm.add_edge("B", "G", head_length=0.08)
    pgm.add_edge("F", "G", head_length=0.08)
    
    return pgm;
%%capture
pgm_bfg_2("images/pgm/bfg-2.png");
Explanation: Example:  Empty Gauge
Now, suppose the gauge reads $G=0$.  We have
$$
P(G=0) = \sum \limits_{B \in {0, 1}} \sum \limits_{F \in {0, 1}} 
 P(G = 0 \mid B, F) P(B) P(F) = 0.315
$$
Verify this!
Example:  Emtpy Gauge
End of explanation
@pgm_render
def pgm_bfg_3():
    pgm = daft.PGM([4, 4], origin=[0, 0])
    # Nodes
    pgm.add_node(daft.Node("G", r"$G$", 2, 3.5, offset=(0, 20), observed=True))
    pgm.add_node(daft.Node("B", r"$B$", 1.3, 2.5, offset=(0, 20), observed=True))
    pgm.add_node(daft.Node("F", r"$F$", 2.7, 2.5, offset=(0, 20)))
    # Add in the edges.
    pgm.add_edge("B", "G", head_length=0.08)
    pgm.add_edge("F", "G", head_length=0.08)
    
    return pgm;
%%capture
pgm_bfg_3("images/pgm/bfg-3.png")
Explanation: Example:  Empty Gauge
Now, we also have 
$$
p(G = 0 \mid F = 0)
= \sum \limits_{B \in {0, 1}} p(G = 0 \mid B, F = 0) p(B) = 0.81
$$ 
Applying Bayes' Rule,
$$
\begin{align}
p(F = 0 \mid G = 0) 
&= \frac{p(G = 0 \mid F = 0) p(F = 0)}{p(G = 0)} \
&\approx 0.257 > p(F = 0) = 0.10
\end{align}
$$
Observing an empty gauge makes it more likely that the tank is empty!
Example:  Emtpy Gauge, Dead Battery
Now, suppose we also observe a dead battery $B =0$.  Then,
$$
\begin{align}
p(F = 0 \mid G = 0, B = 0) 
&= \frac{p(G = 0 \mid B = 0, F = 0) p(F = 0)}{\sum_{F \in {0, 1}} p(G = 0 \mid B = 0, F) p(F)} \
&\approx 0.111
\end{align}
$$
Example:  Emtpy Gauge, Dead Battery
End of explanation
<END_TASK>