<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Validating Likelihood Variance And Single Likelihood Weight Distribution
In this notebook we seek to characterize the noise that appears to accumulate in our learning process and manifests in poor quality posteriors. We narrow our attention to two cases. First, how the number of samples in each halo's likelihood integral impacts the variance of the total likelihood. Second, how the weight contributions of individual halos varies across hypersamples.
Variance Sensitivity to Number of Importance Samples
We take the hyperparameter from the center of the prior distribution (mean of gaussians - [10.709, 0.35899999999999999, 1.1000000000000001, 0.155]) and compute the total likelihood under a number of different random seeds. This allows us to quantify the intrinsic variance in our process. Remember that the return value is the log-likelihood so even a little variance can lead to astronomical differences in weights. Since we plan to use on the order of 10,000 samples we hope to achieve a log-likelihood variance < 2 (so standard deviation of multiplicative weight change is factor of 4 or so).
Step1:  And here we have the standard deviations
Step2:  If we maintain our key assumption from the Quantifying Scaling Accuracy notebook, that the single log-likelihood integrals are independent and identically distributed gaussians. Then with 100 samples, the mean and variance are -8.46 and 0.06 respectively. Importance sampling is precise.
Step3:  Single Likelihood Weights
In this experiment we take 4 hypersamples and collect the weights for the single likelihood corresponding to each halo (the sum of these single log-likelihoods is the total log-likelihood).
Step4:  These look very similar. A natural next thing to check is the correlation between the weight sequences. We see extremely high correlation.
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from matplotlib import rc
rc('text', usetex=True)
!head -n 5 likelihoodvariancetest.txt
multi = np.loadtxt('likelihoodvariancetest.txt')
multi1000 = np.loadtxt('likelihoodvariancetest1000samples.txt')
multi10000 = np.loadtxt('likelihoodvariancetest10000samples.txt')
plt.title('Log-Likelihood Distribution For Fixed Hypersample')
plt.ylabel('Density')
plt.xlabel('Log-Likelihood')
plt.hist(multi[:,4], bins=20, normed=True, alpha=0.5, label='100 samples')
plt.hist(multi1000[:,4], bins=20, normed=True, alpha=0.5, label='1000 samples')
plt.hist(multi10000[:,4], bins=20, normed=True, alpha=0.5, label='10000 samples')
plt.legend(loc=2);
Explanation: Validating Likelihood Variance And Single Likelihood Weight Distribution
In this notebook we seek to characterize the noise that appears to accumulate in our learning process and manifests in poor quality posteriors. We narrow our attention to two cases. First, how the number of samples in each halo's likelihood integral impacts the variance of the total likelihood. Second, how the weight contributions of individual halos varies across hypersamples.
Variance Sensitivity to Number of Importance Samples
We take the hyperparameter from the center of the prior distribution (mean of gaussians - [10.709, 0.35899999999999999, 1.1000000000000001, 0.155]) and compute the total likelihood under a number of different random seeds. This allows us to quantify the intrinsic variance in our process. Remember that the return value is the log-likelihood so even a little variance can lead to astronomical differences in weights. Since we plan to use on the order of 10,000 samples we hope to achieve a log-likelihood variance < 2 (so standard deviation of multiplicative weight change is factor of 4 or so).
End of explanation
print np.std(multi[:,4])
print np.std(multi1000[:,4])
print np.std(multi10000[:,4])
Explanation: And here we have the standard deviations
End of explanation
print 51.0639314938 / len(multi)
print np.mean(multi[:,4]) / 115919
Explanation: If we maintain our key assumption from the Quantifying Scaling Accuracy notebook, that the single log-likelihood integrals are independent and identically distributed gaussians. Then with 100 samples, the mean and variance are -8.46 and 0.06 respectively. Importance sampling is precise.
End of explanation
single = np.zeros((115919,4))
i = j = 0
with open('singleintegralweightvariancetest.txt', 'r') as f:
    for line in f:
        if 'likelihood' in line:
            j += 1
            i = 0
        else:
            single[i,j] = float(line)
            i += 1
f.close()
plt.hist(single[:,0], bins=50, alpha=0.4, label='l1')
plt.hist(single[:,1], bins=50, alpha=0.4, label='l2')
plt.hist(single[:,2], bins=50, alpha=0.4, label='l3')
plt.hist(single[:,3], bins=50, alpha=0.4, label='l4');
plt.title('Single Log-Likelihood Weight Histograms')
plt.ylabel('Count')
plt.xlabel('Log-Likelihood');
Explanation: Single Likelihood Weights
In this experiment we take 4 hypersamples and collect the weights for the single likelihood corresponding to each halo (the sum of these single log-likelihoods is the total log-likelihood).
End of explanation
np.corrcoef(single.transpose())
Explanation: These look very similar. A natural next thing to check is the correlation between the weight sequences. We see extremely high correlation.
End of explanation
<END_TASK>