<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Measures of Dispersion
By Evgenia "Jenny" Nitishinskaya, Maxwell Margenot, and Delaney Granizo-Mackenzie.
Part of the Quantopian Lecture Series
Step1:  Range
Range is simply the difference between the maximum and minimum values in a dataset. Not surprisingly, it is very sensitive to outliers. We'll use numpy's peak to peak (ptp) function for this.
Step2:  Mean Absolute Deviation (MAD)
The mean absolute deviation is the average of the distances of observations from the arithmetic mean. We use the absolute value of the deviation, so that 5 above the mean and 5 below the mean both contribute 5, because otherwise the deviations always sum to 0.
$$ MAD = \frac{\sum_{i=1}^n |X_i - \mu|}{n} $$
where $n$ is the number of observations and $\mu$ is their mean.
Step3:  Variance and standard deviation
The variance $\sigma^2$ is defined as the average of the squared deviations around the mean
Step4:  One way to interpret standard deviation is by referring to Chebyshev's inequality. This tells us that the proportion of samples within $k$ standard deviations (that is, within a distance of $k \cdot$ standard deviation) of the mean is at least $1 - 1/k^2$ for all $k>1$.
Let's check that this is true for our data set.
Step5:  The bound given by Chebyshev's inequality seems fairly loose in this case. This bound is rarely strict, but it is useful because it holds for all data sets and distributions.
Semivariance and semideviation
Although variance and standard deviation tell us how volatile a quantity is, they do not differentiate between deviations upward and deviations downward. Often, such as in the case of returns on an asset, we are more worried about deviations downward. This is addressed by semivariance and semideviation, which only count the observations that fall below the mean. Semivariance is defined as
$$ \frac{\sum_{X_i < \mu} (X_i - \mu)^2}{n_<} $$
where $n_<$ is the number of observations which are smaller than the mean. Semideviation is the square root of the semivariance.
Step6:  A related notion is target semivariance (and target semideviation), where we average the distance from a target of values which fall below that target
<ASSISTANT_TASK:>
Python Code:
# Import libraries
import numpy as np
np.random.seed(121)
# Generate 20 random integers < 100
X = np.random.randint(100, size=20)
# Sort them
X = np.sort(X)
print 'X: %s' %(X)
mu = np.mean(X)
print 'Mean of X:', mu
Explanation: Measures of Dispersion
By Evgenia "Jenny" Nitishinskaya, Maxwell Margenot, and Delaney Granizo-Mackenzie.
Part of the Quantopian Lecture Series:
www.quantopian.com/lectures
github.com/quantopian/research_public
Notebook released under the Creative Commons Attribution 4.0 License.
Dispersion measures how spread out a set of data is. This is especially important in finance because one of the main ways risk is measured is in how spread out returns have been historically. If returns have been very tight around a central value, then we have less reason to worry. If returns have been all over the place, that is risky.
Data with low dispersion is heavily clustered around the mean, while high dispersion a indicates many very large and very small values.
Let's generate an array of random integers to work with.
End of explanation
print 'Range of X: %s' %(np.ptp(X))
Explanation: Range
Range is simply the difference between the maximum and minimum values in a dataset. Not surprisingly, it is very sensitive to outliers. We'll use numpy's peak to peak (ptp) function for this.
End of explanation
abs_dispersion = [np.abs(mu - x) for x in X]
MAD = np.sum(abs_dispersion)/len(abs_dispersion)
print 'Mean absolute deviation of X:', MAD
Explanation: Mean Absolute Deviation (MAD)
The mean absolute deviation is the average of the distances of observations from the arithmetic mean. We use the absolute value of the deviation, so that 5 above the mean and 5 below the mean both contribute 5, because otherwise the deviations always sum to 0.
$$ MAD = \frac{\sum_{i=1}^n |X_i - \mu|}{n} $$
where $n$ is the number of observations and $\mu$ is their mean.
End of explanation
print 'Variance of X:', np.var(X)
print 'Standard deviation of X:', np.std(X)
Explanation: Variance and standard deviation
The variance $\sigma^2$ is defined as the average of the squared deviations around the mean:
$$ \sigma^2 = \frac{\sum_{i=1}^n (X_i - \mu)^2}{n} $$
This is sometimes more convenient than the mean absolute deviation because absolute value is not differentiable, while squaring is smooth, and some optimization algorithms rely on differentiability.
Standard deviation is defined as the square root of the variance, $\sigma$, and it is the easier of the two to interpret because it is in the same units as the observations.
End of explanation
k = 1.25
dist = k*np.std(X)
l = [x for x in X if abs(x - mu) <= dist]
print 'Observations within', k, 'stds of mean:', l
print 'Confirming that', float(len(l))/len(X), '>', 1 - 1/k**2
Explanation: One way to interpret standard deviation is by referring to Chebyshev's inequality. This tells us that the proportion of samples within $k$ standard deviations (that is, within a distance of $k \cdot$ standard deviation) of the mean is at least $1 - 1/k^2$ for all $k>1$.
Let's check that this is true for our data set.
End of explanation
# Because there is no built-in semideviation, we'll compute it ourselves
lows = [e for e in X if e <= mu]
semivar = np.sum( (lows - mu) ** 2 ) / len(lows)
print 'Semivariance of X:', semivar
print 'Semideviation of X:', np.sqrt(semivar)
Explanation: The bound given by Chebyshev's inequality seems fairly loose in this case. This bound is rarely strict, but it is useful because it holds for all data sets and distributions.
Semivariance and semideviation
Although variance and standard deviation tell us how volatile a quantity is, they do not differentiate between deviations upward and deviations downward. Often, such as in the case of returns on an asset, we are more worried about deviations downward. This is addressed by semivariance and semideviation, which only count the observations that fall below the mean. Semivariance is defined as
$$ \frac{\sum_{X_i < \mu} (X_i - \mu)^2}{n_<} $$
where $n_<$ is the number of observations which are smaller than the mean. Semideviation is the square root of the semivariance.
End of explanation
B = 19
lows_B = [e for e in X if e <= B]
semivar_B = sum(map(lambda x: (x - B)**2,lows_B))/len(lows_B)
print 'Target semivariance of X:', semivar_B
print 'Target semideviation of X:', np.sqrt(semivar_B)
Explanation: A related notion is target semivariance (and target semideviation), where we average the distance from a target of values which fall below that target:
$$ \frac{\sum_{X_i < B} (X_i - B)^2}{n_{<B}} $$
End of explanation
<END_TASK>