<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Normal distribution
Standard normal distribution takes a bell curve. It is also called as gaussian distribution. Values in nature are believed to take a normal distribution. The equation for normal distribution is 
$$y = \frac{1}{\sigma\sqrt{2\pi}} e^{\frac{(x-\mu)^2}{2\sigma^2}}$$
where
$\mu$ is mean
$\sigma$ is standard deviation
$\pi$ = 3.14159..
$e$ = 2.71828.. (natural log)
Step1:  The above is the standard normal distribution. Its mean is 0 and SD is 1. About 95% values fall within $\mu \pm 2 SD$ and 98% within $\mu \pm 3 SD$
The area under this curve is 1 which gives the probability of values falling within the range of standard normal.
A common use is to find the probability of a value falling at a particular range. For instance, find $p(-2 \le z \le 2)$ which is the probability of a value falling within $\mu \pm 2SD$. This calculated by summing the area under the curve between these bounds.
$$p(-2 \le z \le 2) = 0.9544$$ which is 95.44% probability. Its z score is 0.9544.
Similarly
$$p(z \ge 5.1) = 0.00000029$$
Finding z score and p values using SciPy
The standard normal is useful as a z table to look up the probability of a z score (x axis). You can use Scipy to accomplish this.
Levels of significance
By rule of thumb, a z score greater than 0.005 is considered significant as such a value has a very low probability of occuring. Thus, there is less chance of it occurring randomly and hence, there is probably a force acting on it (significant force, not random chance).
Transformation to standard normal
If the distribution of a phenomena follows normal dist, then you can transform it to standard normal, so you can measure the z scores. To do so,
$$std normal value = \frac{observed - \mu}{\sigma}$$ You subtract the mean and divide by SD of the distribution.
Example
Let X be age of US presidents at inaugration. $X \in N(\mu = 54.8, \sigma=6.2)$. What is the probability of choosing a president at random that is less than 44 years of age.
We need to find $p(x<44)$. First we need to transform to standard normal.
$$p(z< \frac{44-54.8}{6.2})$$
$$p(z<-1.741) = 0.0409 \approx 4\%$$
Finding z score and p values using SciPy
The standard normal is useful as a z table to look up the probability of a z score (x axis). You can use Scipy to accomplish this.
Step2:  Let us try for some common z scores
Step3:  As you noticed, the norm.cdf() function gives the cumulative probability (left tail) from -3 to 3 approx. If you need right tailed distribution, you simply subtract this value from 1.
Finding z score from a p value
Sometimes, you have the probability (p value), but want to find the z score or how many SD does this value fall from mean. You can do this inverse using ppf().
Step4:  As is the ppf() function gives only positive z scores, you need to apply $\pm$ to it.
Transformation to standard normal and machine learning
Transforming features to standard normal has applications in machine learning. As each feature has a different unit, their range, standard deviation vary. Hence we scale them all to standard normal distribution with mean=0 and SD=1. This way a learner finds those variables that are truly influencial and not simply because it has a larger range.
To accomplish this easily, we use scikit-learn's StandardScaler object as shown below
Step5:  Now let us use scikit-learn to easily transform this dataset
Step6:  As you see above, the shape of distribution is the same, just the values are scaled.
Assessing normality of a distribution
To assess how normal a distribution of values is, we sort the values, then plot them against sorted values of standard normal distribution. If the values fall on a straight line, then they are normally distributed, else they exhibit skewness and or kurtosis.
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
vals = np.random.standard_normal(100000)
len(vals)
fig, ax = plt.subplots(1,1)
hist_vals = ax.hist(vals, bins=200, color='red', density=True)
Explanation: Normal distribution
Standard normal distribution takes a bell curve. It is also called as gaussian distribution. Values in nature are believed to take a normal distribution. The equation for normal distribution is 
$$y = \frac{1}{\sigma\sqrt{2\pi}} e^{\frac{(x-\mu)^2}{2\sigma^2}}$$
where
$\mu$ is mean
$\sigma$ is standard deviation
$\pi$ = 3.14159..
$e$ = 2.71828.. (natural log)
End of explanation
import scipy.stats as st
# compute the p value for a z score
st.norm.cdf(-1.741)
Explanation: The above is the standard normal distribution. Its mean is 0 and SD is 1. About 95% values fall within $\mu \pm 2 SD$ and 98% within $\mu \pm 3 SD$
The area under this curve is 1 which gives the probability of values falling within the range of standard normal.
A common use is to find the probability of a value falling at a particular range. For instance, find $p(-2 \le z \le 2)$ which is the probability of a value falling within $\mu \pm 2SD$. This calculated by summing the area under the curve between these bounds.
$$p(-2 \le z \le 2) = 0.9544$$ which is 95.44% probability. Its z score is 0.9544.
Similarly
$$p(z \ge 5.1) = 0.00000029$$
Finding z score and p values using SciPy
The standard normal is useful as a z table to look up the probability of a z score (x axis). You can use Scipy to accomplish this.
Levels of significance
By rule of thumb, a z score greater than 0.005 is considered significant as such a value has a very low probability of occuring. Thus, there is less chance of it occurring randomly and hence, there is probably a force acting on it (significant force, not random chance).
Transformation to standard normal
If the distribution of a phenomena follows normal dist, then you can transform it to standard normal, so you can measure the z scores. To do so,
$$std normal value = \frac{observed - \mu}{\sigma}$$ You subtract the mean and divide by SD of the distribution.
Example
Let X be age of US presidents at inaugration. $X \in N(\mu = 54.8, \sigma=6.2)$. What is the probability of choosing a president at random that is less than 44 years of age.
We need to find $p(x<44)$. First we need to transform to standard normal.
$$p(z< \frac{44-54.8}{6.2})$$
$$p(z<-1.741) = 0.0409 \approx 4\%$$
Finding z score and p values using SciPy
The standard normal is useful as a z table to look up the probability of a z score (x axis). You can use Scipy to accomplish this.
End of explanation
[st.norm.cdf(-3), st.norm.cdf(-1), st.norm.cdf(0), st.norm.cdf(1), st.norm.cdf(2)]
Explanation: Let us try for some common z scores:
End of explanation
# Find Z score for a probability of 0.97 (2sd)
st.norm.ppf(0.97)
[st.norm.ppf(0.95), st.norm.ppf(0.97), st.norm.ppf(0.98), st.norm.ppf(0.99)]
Explanation: As you noticed, the norm.cdf() function gives the cumulative probability (left tail) from -3 to 3 approx. If you need right tailed distribution, you simply subtract this value from 1.
Finding z score from a p value
Sometimes, you have the probability (p value), but want to find the z score or how many SD does this value fall from mean. You can do this inverse using ppf().
End of explanation
demo_dist = 55 + np.random.randn(200) * 3.4
std_normal = np.random.randn(200)
[demo_dist.mean(), demo_dist.std(), demo_dist.min(), demo_dist.max()]
[std_normal.mean(), std_normal.std(), std_normal.min(), std_normal.max()]
Explanation: As is the ppf() function gives only positive z scores, you need to apply $\pm$ to it.
Transformation to standard normal and machine learning
Transforming features to standard normal has applications in machine learning. As each feature has a different unit, their range, standard deviation vary. Hence we scale them all to standard normal distribution with mean=0 and SD=1. This way a learner finds those variables that are truly influencial and not simply because it has a larger range.
To accomplish this easily, we use scikit-learn's StandardScaler object as shown below:
End of explanation
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
demo_dist = demo_dist.reshape(200,1)
demo_dist_scaled = scaler.fit_transform(demo_dist)
[round(demo_dist_scaled.mean(),3), demo_dist_scaled.std(), demo_dist_scaled.min(), demo_dist_scaled.max()]
fig, axs = plt.subplots(2,2, figsize=(15,8))
p1 = axs[0][0].scatter(sorted(demo_dist), sorted(std_normal))
axs[0][0].set_title("Scatter of original dataset against standard normal")
p1 = axs[0][1].scatter(sorted(demo_dist_scaled), sorted(std_normal))
axs[0][1].set_title("Scatter of scaled dataset against standard normal")
p2 = axs[1][0].hist(demo_dist, bins=50)
axs[1][0].set_title("Histogram of original dataset against standard normal")
p3 = axs[1][1].hist(demo_dist_scaled, bins=50)
axs[1][1].set_title("Histogram of scaled dataset against standard normal")
Explanation: Now let us use scikit-learn to easily transform this dataset
End of explanation
demo_dist = 55 + np.random.randn(200) * 3.4
std_normal = np.random.randn(200)
demo_dist = sorted(demo_dist)
std_normal = sorted(std_normal)
plt.scatter(demo_dist, std_normal)
Explanation: As you see above, the shape of distribution is the same, just the values are scaled.
Assessing normality of a distribution
To assess how normal a distribution of values is, we sort the values, then plot them against sorted values of standard normal distribution. If the values fall on a straight line, then they are normally distributed, else they exhibit skewness and or kurtosis.
End of explanation
<END_TASK>