<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Lecture 15
Step1:  Some very straightforward statistics are the number of data points, the largest value, and the smallest value. These shouldn't be immediately ignored, but they are of limited utility.
Step2:  Mean
Step3:  The mean is very simple to compute
Step4:  The mean is sensitive to outliers, meaning one or two data points that lie well beyond all the others can disproportionately affect the value of the mean. In the above simple example, the lone outlier of 38 pulls the mean to be larger than all the other data points except for 38; not exactly a representative sample!
Median
Step5:  The median is computed by sorting all your data and picking out the middle value (or averaging the two middle data points, if you have an even amount of data). In this way, the median does not directly depend on the vast majority of your data; just on whatever value(s) happen to be in the middle. It is also not trivial to compute
Step6:  Second-order statistics
Second-order statistics rely both on the data itself, and first-order statistics, to compute.
Variance
Step7:  The variance is computed by subtracting each individual data point from the average of the whole data set, squaring this difference, and summing all these differences together before finally dividing by the number of data points.
Variance may not be familiar, but you've probably heard of its relative
Step8:  Like the mean, variance (and standard deviation) uses all the data points to compute, and is therefore sensitive to outliers.
Interquartile Range
Step9:  This, like the median, is robust to outliers. But also like the median, it relies on sorting the data first, then picking out the value 1/4 of the way down the dataset and subtracting it from the value 3/4 of the way down the dataset. This can be expensive in large datasets.
Summary so far
So far, we've looked at ways of quantitatively summarizing datasets. These involve first and second-order statistics, including
Mean
Step10:  It's tough to see, isn't it? Let's try 1000 observations.
Step11:  That looks a little better! Maybe 10,000 data points, just for grins?
Step12:  There's the bell curve we know and love!
So what happened?
We had some observations $x$--a regular array with a bunch of numbers in it (10,000 by the end, to be exact). In fact, here's what the array looked like
Step13:  This could be any old dataset! In fact, forget for a moment that we generated this dataset ourselves, and instead think that this could be a dataset we picked up from the web.
We're able to compute some statistics from it
Step14:  You'll notice the mean is very close to 0, and the variance is likewise very close to 1. Since we ourselves set the mean and variance for the random number generator, we know that these are very close to the true mean and true variance, but in general we wouldn't necessarily know that.
Instead, we'd have computed the sample mean and sample variance from the data, as we did above, and then assumed the data to be Gaussian; after all, it certainly looks like a bell curve!
Distributions
This brings us to distributions. When we made those histograms of the observations, we were creating a distribution of the data. The mean and variance statistics are parameters that govern the shape of those distributions. Take a look at how the distribution changes when we change its parameters
<ASSISTANT_TASK:>
Python Code:
import numpy as np
np.random.seed(29384924)
data = np.random.randint(10, size = 100)
print(data)
Explanation: Lecture 15: Introduction to Probability and Statistics
CSCI 1360E: Foundations for Informatics and Analytics
Overview and Objectives
To wrap up this week's introduction into the fundamental tools of data science, today we'll jump into probability and statistics. This is yet another topic that could span an entire course, and like linear algebra I highly recommend taking a stats course at some point. By the end of this lecture, you should be able to
Define probability and statistics
Understand the roles of random variables and parametric distributions
Compute basic first and second-order statistics on datasets
Explain conditional probability and its role in Bayes' Theorem
Part 1: Statistics
One definition of statistics is that it is a way of summarizing data. 
Another definition is that statistics is making predictions on the whole population from a sample of the population.
Of course, these definitions are gross oversimplifications, but I'll illustrate with an example.
Example: Giving a Talk
You're an intern at the Large Hadron Collider (LHC), and you've been instrumental in devising the experiments that seem to have uncovered hard evidence for the existence of the Higgs Boson. This evidence consists of the results of billions of subatomic collisions taking place deep in the bowels of the LHC.
Some high-level executives from the funding agencies responsible for keeping the lights on at the LHC want to hear about these exciting results, but all your immediate bosses are indisposed and have asked you to step in.
How do you present the findings? Do you provide a raw list of the billions of collisions? Do you take some averages of the data? How many? How representative of the full billion data points are these handful of averages? Can you quantify the uncertainty in these averages, i.e. how far off they could potentially be?
First-order statistics
First-order statistics are summaries of the data that rely only on the data itself.
End of explanation
print("Number of data points: {}".format(data.shape[0]))
print("Largest value: {}".format(data.max()))
print("Smallest value: {}".format(data.min()))
Explanation: Some very straightforward statistics are the number of data points, the largest value, and the smallest value. These shouldn't be immediately ignored, but they are of limited utility.
End of explanation
print(data.mean())  # Mean
Explanation: Mean: You've seen this all the time--it's the average of all the data!
End of explanation
outlier = np.array([1, 1, 2, 3, 2, 1, 3, 2, 38])  # Note the outlier of 38 at the end.
print(outlier.mean())
Explanation: The mean is very simple to compute: it's the sum of the data divided by the number of data points. The mean depends at least a little bit on every single data point. This can be advantageous in certain situations, as it varies smoothly as more data points are added or some are removed.
However, this property can also be problematic. A famous example that exploits this weakness is that in the mid-1980s, the major at UNC with the highest average starting salary was geography...however, this was mostly on account of a particular NBA star by the name of Michael Jordan.
End of explanation
print(data)
print(np.median(data))
Explanation: The mean is sensitive to outliers, meaning one or two data points that lie well beyond all the others can disproportionately affect the value of the mean. In the above simple example, the lone outlier of 38 pulls the mean to be larger than all the other data points except for 38; not exactly a representative sample!
Median: The "middle" data point.
End of explanation
print(outlier)
print(np.median(outlier))
Explanation: The median is computed by sorting all your data and picking out the middle value (or averaging the two middle data points, if you have an even amount of data). In this way, the median does not directly depend on the vast majority of your data; just on whatever value(s) happen to be in the middle. It is also not trivial to compute: you have to sort your data first, which might be tricky if you have billions of data points.
On the other hand, the median is not sensitive to outliers. In fact, it's robust to outliers, meaning it wholesale ignores them.
End of explanation
print(data)
print(data.var())
Explanation: Second-order statistics
Second-order statistics rely both on the data itself, and first-order statistics, to compute.
Variance: This measures how spread out your data are. More specifically, it measures how much your data varies from its mean.
End of explanation
print(np.sqrt(data.var()))
print(data.std())
Explanation: The variance is computed by subtracting each individual data point from the average of the whole data set, squaring this difference, and summing all these differences together before finally dividing by the number of data points.
Variance may not be familiar, but you've probably heard of its relative: standard deviation is just the square root of the variance!
End of explanation
print(np.percentile(data, 75) - np.percentile(data, 25))
Explanation: Like the mean, variance (and standard deviation) uses all the data points to compute, and is therefore sensitive to outliers.
Interquartile Range: The difference between the 75% percentile and 25% of the data.
End of explanation
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
# Observe 100 data points from a Gaussian random variable with mean = 0.0 and variance = 1.0.
observations = np.random.normal(size = 100)
plt.hist(observations)
Explanation: This, like the median, is robust to outliers. But also like the median, it relies on sorting the data first, then picking out the value 1/4 of the way down the dataset and subtracting it from the value 3/4 of the way down the dataset. This can be expensive in large datasets.
Summary so far
So far, we've looked at ways of quantitatively summarizing datasets. These involve first and second-order statistics, including
Mean: average of the data. Easy to compute, but sensitive to outliers.
Median: "middle" data point. Harder to compute, but robust to outliers.
Variance (and standard deviation): squared "spread" of the data; a measure of how tightly-bunched (or not) around the mean the data are. Easy to compute, but sensitive to outliers.
Interquartile Range: difference between the 25th and 75th percentiles of the data. Harder to compute, but robust to outliers.
Part 2: Random Variables and Distributions
These methods for describing and summarizing data will come in handy as we deal with random variables.
A random variable is, first and foremost, a variable: we don't know its value. It is often represented by a capital letter, e.g. $X$.
While we can't know the exact value of a random variable, we often have a handful of observations of the variable--usually denoted with a lowercase version of the corresponding capital letter, e.g. $x$--and it's from these observations that we compute things like mean and variance in order to describe the random variable itself.
Is this all coming together, or what?!
As a result of these observations $x_1, x_2, ..., x_n$ of the random variable $X$, we can usually say something about how $X$ is distributed.
End of explanation
# Observe **1000** data points from a Gaussian random variable with mean = 0.0 and variance = 1.0.
observations = np.random.normal(size = 1000)
plt.hist(observations)
Explanation: It's tough to see, isn't it? Let's try 1000 observations.
End of explanation
# Observe **** 10,000 **** data points from a Gaussian random variable with mean = 0.0 and variance = 1.0.
observations = np.random.normal(size = 10000)
_, _, _ = plt.hist(observations, bins = 25)
Explanation: That looks a little better! Maybe 10,000 data points, just for grins?
End of explanation
print(observations)
Explanation: There's the bell curve we know and love!
So what happened?
We had some observations $x$--a regular array with a bunch of numbers in it (10,000 by the end, to be exact). In fact, here's what the array looked like:
End of explanation
print("Mean: {:.2f}".format(observations.mean()))
print("Variance: {:.2f}".format(observations.var()))
Explanation: This could be any old dataset! In fact, forget for a moment that we generated this dataset ourselves, and instead think that this could be a dataset we picked up from the web.
We're able to compute some statistics from it:
End of explanation
from scipy.stats import norm
xs = np.linspace(-5, 5, 100)
plt.plot(xs, norm.pdf(xs, loc = 0, scale = 1), '-', label = "mean=0, var=1")
plt.plot(xs, norm.pdf(xs, loc = 0, scale = 2), '--', label = "mean=0, var=2")
plt.plot(xs, norm.pdf(xs, loc = 0, scale = 0.5), ':', label = "mean=0, var=0.5")
plt.plot(xs, norm.pdf(xs, loc = -1, scale = 1), '-.', label = "mean=-1, var=1")
plt.legend(loc = 0)
plt.title("Various Normal Distributions")
Explanation: You'll notice the mean is very close to 0, and the variance is likewise very close to 1. Since we ourselves set the mean and variance for the random number generator, we know that these are very close to the true mean and true variance, but in general we wouldn't necessarily know that.
Instead, we'd have computed the sample mean and sample variance from the data, as we did above, and then assumed the data to be Gaussian; after all, it certainly looks like a bell curve!
Distributions
This brings us to distributions. When we made those histograms of the observations, we were creating a distribution of the data. The mean and variance statistics are parameters that govern the shape of those distributions. Take a look at how the distribution changes when we change its parameters:
End of explanation
<END_TASK>