<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 3 bits
In the below cells I explore the next step - three hidden units and three visible units.
The results are very good, for three hiddens and three visibles it works as expected. However it needs a good model to behave itself. Hence I may run build_and_eval several times until all the expected reconstructions happen almost equally.
Step1:  2 Hiddens, 3 Visible
I Believe I should be able to create an rbm with this configuration, however all reconstructions have been mixed so far.
Step2:  4 bits #
Lets try 4 hiddens and 4 visibles
- Excellent it works really well. See graphs below
Step3:  Five Bits
Let's look at whether it is still scaling with 5 bits, (5hid and 5vis)
Step4:  Awesome,
So it is working up to 5 bits. it just needs a good model - which makes sense. 
Okay lets try 5 hiddens and 4 visibles
I would expect this to still perform well, that is separate the causes. the model should be really good, and therefore it should do a great job
Step5:  Lets try and make a really, really good model
I'm going to do the pathological number of hidden units, that is $2n$ for $n$ visibles. Can I train a perfect (likely super overfit model) and examine the partitioning affect.
Step6:  That is a really strong model
I would expect 50% of the patterns to be [1,0] and visa versa [0,1]
It took a few tries to get here though
Step8:  Not bad. It separates quite well.
Step9:  Updating the Hidden States Layerwise V Alternatives
So we know that we should possibly (but hopefully not) actually let a given update to the hiddens mutilple times, because the $h^A$ and $h^A$ are depedant on each other. Actaully each unit of each hidden layer is depedant on each other. So the full version would be to update a given $h^A_j$ for all $j$ and do the same for $h^B_k $ - Realistically do this I would probably swap to a Java implementation, two reasons
Step10:  hmmmm roughly the same - Good
So now lets try with a worse model, does the new approach account for a weaker model. I don't think it will, but it's worth seeing.
<ASSISTANT_TASK:>
Python Code:
# model = build_and_eval(3,3,epochs)
b = BernoulliRBM(n_components=3,n_iter=10000,learning_rate=0.02)
b.fit(np.eye(3))
# b.gibbs(np.array([1,0,0]))
# model.weights
model.hidden_bias
model.visible_bias
b
vs = [np.array([1,1,0]),np.array([0,1,1]),np.array([1,0,0])]
for v in vs:
    eval_partitioned(model,v)
Explanation: 3 bits
In the below cells I explore the next step - three hidden units and three visible units.
The results are very good, for three hiddens and three visibles it works as expected. However it needs a good model to behave itself. Hence I may run build_and_eval several times until all the expected reconstructions happen almost equally.
End of explanation
two_bit = build_and_eval(2,3, epochs= 10000, n_reconstructions=10000)
vs = [np.array([1,1,0]),np.array([0,1,1]),np.array([1,0,0])]
for v in vs:
    eval_partitioned(two_bit,v)
Explanation: 2 Hiddens, 3 Visible
I Believe I should be able to create an rbm with this configuration, however all reconstructions have been mixed so far.
End of explanation
four_bit = build_and_eval(4,4)
vs = [np.array([1,1,0,0]),np.array([0,1,0,1]),np.array([1,0,0,1])]
for v in vs:
    eval_partitioned(four_bit,v)
Explanation: 4 bits #
Lets try 4 hiddens and 4 visibles
- Excellent it works really well. See graphs below
End of explanation
five_bit = build_and_eval(5,5)
eval_partitioned(five_bit,np.array([1,1,0,0,0]))
Explanation: Five Bits
Let's look at whether it is still scaling with 5 bits, (5hid and 5vis)
End of explanation
five_hid_four_vis = build_and_eval(5,4)
vs = [np.array([1,1,0,0]),np.array([0,1,0,1]),np.array([1,0,0,1])]
for v in vs:
    eval_partitioned(five_hid_four_vis,v)
four_hid_five_vis = build_and_eval(4,5)
d = eval_partitioned(four_hid_five_vis,np.array([0,1,0,1,0]), times=5000)
d.highest_n_items(3)
Explanation: Awesome,
So it is working up to 5 bits. it just needs a good model - which makes sense. 
Okay lets try 5 hiddens and 4 visibles
I would expect this to still perform well, that is separate the causes. the model should be really good, and therefore it should do a great job
End of explanation
model = build_and_eval(n_hid= 4, n_vis= 2,epochs=1000)
Explanation: Lets try and make a really, really good model
I'm going to do the pathological number of hidden units, that is $2n$ for $n$ visibles. Can I train a perfect (likely super overfit model) and examine the partitioning affect.
End of explanation
eval_partitioned(model, np.array([1,1]), times=10000)
Explanation: That is a really strong model
I would expect 50% of the patterns to be [1,0] and visa versa [0,1]
It took a few tries to get here though
End of explanation
class PlottingSampler(ApproximatedSampler):
    Wrapper for the sampler allow us to observe the calculation
    def approx_correction(self, h_a, h_b, w_a, w_b,v):
        c_a, c_b = super().approx_correction(h_a,h_b,w_a, w_b,v) # wrap those delicious corrections up
#         print("C_A{}\nC_B{}".format(c_a, c_b))
        self.plot_all(self.plot_ready_shape(h_a),self.plot_ready_shape(h_b),c_a,c_b)
        plt.show()
        return c_a, c_b
    def plot_ready_shape(self, h):
        return h.reshape(1,h.shape[0])
    
    def plot_all(self,h_a,h_b,c_a, c_b):
        plt.subplot(141)
        self.plot_hid('h_a',h_a)
        plt.subplot(142)     
        self.plot_hid('h_b',h_b)
        plt.subplot(143)
        self.plot_correction('c_a',self.reshape_for_plot(c_a.sum(1)))
        plt.subplot(144)     
        self.plot_correction('c_b',self.reshape_for_plot(c_b.sum(1)))
        plt.colorbar(ticks=[-1,0,1])
    
    def reshape_for_plot(self, c):
        return c.reshape(1,c.shape[0])
    
    def plot_correction(self,title, c):
        plt.title(title)
        plt.axis('off')
        plt.imshow(c, cmap='seismic', interpolation ='nearest')
    def plot_hid(self,title,h):
        plt.title(title)
        plt.axis('off')
        plt.imshow(h, cmap='copper', interpolation ='nearest')
    
Explanation: Not bad. It separates quite well.
End of explanation
## Separation with 2 hids and 2 vis
model = build_and_eval(2,2)
n_hid = model.num_hid()
h_a = np.random.randint(2,size=(n_hid,))
h_b = np.random.randint(2,size=(n_hid,))
v=np.array([1,1])
s = ApproximatedSampler(model.weights, model.weights,model.hidden_bias, model.hidden_bias)
plotter.plot_dict(perform(h_a, h_b, v, s, 1000, 400), title="v'| {}".format(v), size = "")
# Compare that to the separation of the layerwise h_a updater
s = LayerWiseApproxSampler(model.weights, model.weights,model.hidden_bias, model.hidden_bias)
plotter.plot_dict(perform(h_a, h_b, v, s, 1000, 400), title="v'| {}".format(v), size = "")
Explanation: Updating the Hidden States Layerwise V Alternatives
So we know that we should possibly (but hopefully not) actually let a given update to the hiddens mutilple times, because the $h^A$ and $h^A$ are depedant on each other. Actaully each unit of each hidden layer is depedant on each other. So the full version would be to update a given $h^A_j$ for all $j$ and do the same for $h^B_k $ - Realistically do this I would probably swap to a Java implementation, two reasons:
- Efficiency, for $1,..,n $ hidden units I would have $n$ iterations for each $j$ plus the eqiuvalent $k$ for the the other layer.. :(
- Indices, personally a lot easier to keep track of in Java. Would want typechecking and explicit for loops.
Hopefully it won't come to that, hence what i'm doing below
Up until this point I update the hidden states (for both layers) all at once. I calculate $h^A$ and $h^B$ all at once given a $v$. But it's worth looking into how this affects the reconstructions. So I will compare the current technique to:
- Update $h^A | v$ to get $h'^A$, where the correction is calculated using $h^B$
- Update $h^B | v$ to get $h'^B$, where the correction is now calculated using $h'^A$
- Update $h^A | v$ to get $h'^A$, where the correction is calculated using $h'^A$
- Finally do an update given $h'^A$ and $h'^A$
Or in words: Update the left layer, then update the right based on the update to the left. Then update the left based on the new right and the current left. And then update both the left and right based on these values. Hopefully this won't perform drasitcally better, especially because I am only Markov Chaining(?Is this what I am doing?)
End of explanation
# Two hiddens
model = build_and_eval(2,3)
n_hid = model.num_hid()
h_a = np.random.randint(2,size=(n_hid,))
h_b = np.random.randint(2,size=(n_hid,))
v=np.array([1,1,0])
s = LayerWiseApproxSampler(model.weights, model.weights,model.hidden_bias, model.hidden_bias)
plotter.plot_dict(perform(h_a, h_b, v, s, 1000, 1000), title="v'| {}".format(v), size = "")
model = build_and_eval(2,3, epochs= 20000,n_gibbs_p_r= 400 )
n_hid = model.num_hid()
h_a = np.random.randint(2,size=(n_hid,))
h_b = np.random.randint(2,size=(n_hid,))
v=np.array([1,1,0])
s = LayerWiseApproxSampler(model.weights, model.weights,model.hidden_bias, model.hidden_bias)
plotter.plot_dict(perform(h_a, h_b, v, s, 1000, 1000), title="v'| {}".format(v), size = "")
Explanation: hmmmm roughly the same - Good
So now lets try with a worse model, does the new approach account for a weaker model. I don't think it will, but it's worth seeing.
End of explanation
<END_TASK>