<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Module 8
Step1:  The tricky histogram with pre-counted data
Let's revisit the table from the class
| Hours | Frequency |
|-------|-----------|
| 0-1   | 4,300     |
| 1-3   | 6,900     |
| 3-5   | 4,900     |
| 5-10  | 2,000     |
| 10-24 | 2,100     |
You can draw a histogram by just providing bins and counts instead of a list of numbers. So, let's try that.
Step2:  Q
Step3:  As you can see, the default histogram does not normalize with binwidth and simply shows the counts! This can be very misleading if you are working with variable bin width (e.g. logarithmic bins). So please be mindful about histograms when you work with variable bins. 
Q
Step4:  Let's use an actual dataset
Step5:  Note
Step6:  Let's plot the histogram of IMDB ratings.
Step7:  Did you get an error or a warning? What's going on? 
The problem is that the column contains NaN (Not a Number) values, which represent missing data points. The following command check whether each value is a NaN and returns the result.
Step8:  As you can see there are a bunch of missing rows. You can count them.
Step9:  or drop them.
Step10:  The dropna can be applied to the dataframe too. 
Q
Step11:  How does matplotlib decides the bins? Actually matplotlib's hist function uses numpy's histogram function under the hood. 
Q
Step12:  Have you noticed that this function returns three objects? Take a look at the documentation here to figure out what they are.
To get the returned three objects
Step13:  Here, n_raw contains the values of histograms, i.e., the number of movies in each of the 10 bins. Thus, the sum of the elements in n_raw should be equal to the total number of movies. 
Q
Step14:  The second returned object (bins_raw) is a list containing the edges of the 10 bins
Step15:  The width is same as the maximum value minus minimum value, divided by 10.
Step16:  Now, let's plot a normalized (density) histogram.
Step17:  The ten bins do not change. But now n represents the density of the data inside each bin. In other words, the sum of the area of each bar will equal to 1. 
Q
Step18:  Anyway, these data generated from the hist function is calculated from numpy's histogram function. https
Step19:  If you look at the documentation, you can see that numpy uses simply 10 as the default number of bins. But you can set it manually or set it to be auto, which is the "Maximum of the sturges and fd estimators.". Let's try this auto option.
Step20:  Consequences of the binning parameter
Let's explore the effect of bin size using small multiples. In matplotlib, you can use subplot to put multiple plots into a single figure. 
For instance, you can do something like
Step21:  What does the argument in plt.subplot(1,2,1) mean? If you're not sure, check out
Step22:  Do you see the issues with having too few bins or too many bins? In particular, do you notice weird patterns that emerge from bins=30? 
Q
Step23:  Formulae for choosing the number of bins.
We can manually choose the number of bins based on those formulae.
Step24:  But we can also use built-in formulae too. Let's try all of them.
Step25:  Some are decent, but several of them tend to overestimate the good number of bins. As you have more data points, some of the formulae may overestimate the necessary number of bins. Particularly in our case, because of the precision issue, we shouldn't increase the number of bins too much.
Then, how should we choose the number of bins?
So what's the conclusion? use Scott's rule or Sturges' formula? 
No, I think the take-away is that you should understand how the inappropriate number of bins can mislead you and you should try multiple number of bins to obtain the most accurate picture of the data. Although the 'default' may work in most cases, don't blindly trust it! Don't judge the distribution of a dataset based on a single histogram. Try multiple parameters to get the full picture!
CDF (Cumulative distribution function)
Drawing a CDF is easy. Because it's very common data visualization, histogram has an option called cumulative.
Step26:  You can also combine with options such as histtype and density.
Step27:  And increase the number of bins.
Step28:  This method works fine. By increasing the number of bins, you can get a CDF in the resolution that you want. But let's also try it manually to better understand what's going on. First, we should sort all the values.
Step29:  We need to know the number of data points,
Step30:  And I think this may be useful for you.
Step31:  Q
Step32:  A bit more histogram with altair
As you may remember, you can get a pandas dataframe from vega_datasets package and use it to create visualizations. But, if you use altair, you can simply pass the URL instead of the actual data.
Step33:  As mentioned before, in altair histogram is not special. It is just a plot that use bars (mark_bar()) where X axis is defined by IMDB_Rating with bins (bin=True), and Y axis is defined by count() aggregation function.
Step34:  Have you noted that it is IMDB_Rating
Step35:  In altair, you want to specify the data types using one of the four categories
Step36:  Composing charts in altair
altair has a very nice way to compose multiple plots. Two histograms side by side? just do the following.
Step37:  Vertical commposition?
Step38:  Shall we avoid some repetitions? You can define a base empty chart first and then assign encodings later when you put together multiple charts together. Here is an example
Step39:  Q
<ASSISTANT_TASK:>
Python Code:
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import altair as alt
import pandas as pd
import matplotlib
matplotlib.__version__
Explanation: Module 8: Histogram and CDF
A deep dive into Histogram and boxplot.
End of explanation
bins = [0, 1, 3, 5, 10, 24]
data = {0.5: 4300, 2: 6900, 4: 4900, 7: 2000, 15: 2100} 
data.keys()
Explanation: The tricky histogram with pre-counted data
Let's revisit the table from the class
| Hours | Frequency |
|-------|-----------|
| 0-1   | 4,300     |
| 1-3   | 6,900     |
| 3-5   | 4,900     |
| 5-10  | 2,000     |
| 10-24 | 2,100     |
You can draw a histogram by just providing bins and counts instead of a list of numbers. So, let's try that.
End of explanation
# TODO: draw a histogram with weighted data. 
Explanation: Q: Draw histogram using this data. Useful query: Google search: matplotlib histogram pre-counted
End of explanation
# TODO: fix it with density option. 
Explanation: As you can see, the default histogram does not normalize with binwidth and simply shows the counts! This can be very misleading if you are working with variable bin width (e.g. logarithmic bins). So please be mindful about histograms when you work with variable bins. 
Q: You can fix this by using the density option.
End of explanation
import vega_datasets
vega_datasets.__version__
Explanation: Let's use an actual dataset
End of explanation
movies = vega_datasets.data.movies()
movies.head()
Explanation: Note: Please check your vega_datasets version using vega_datasets.__version__. If you have a version lower than 0.9.0, you will need to check the column names in movies.head() and update it accordingly in the code cells below.
End of explanation
try:
    plt.hist(movies['IMDB_Rating'])
except KeyError as e:
    print("movies has given KeyError: ", str(e))
Explanation: Let's plot the histogram of IMDB ratings.
End of explanation
movies['IMDB_Rating'].isna()
Explanation: Did you get an error or a warning? What's going on? 
The problem is that the column contains NaN (Not a Number) values, which represent missing data points. The following command check whether each value is a NaN and returns the result.
End of explanation
sum(movies['IMDB_Rating'].isna())
Explanation: As you can see there are a bunch of missing rows. You can count them.
End of explanation
IMDB_ratings_nan_dropped = movies['IMDB_Rating'].dropna()
len(IMDB_ratings_nan_dropped)
213 + 2988
Explanation: or drop them.
End of explanation
# TODO
# Both should be zero. 
print(sum(movies['IMDB_Rating'].isna()), sum(movies['IMDB_Votes'].isna()))
Explanation: The dropna can be applied to the dataframe too. 
Q: drop rows from movies dataframe where either IMDB_Rating or IMDB_Votes is NaN.
End of explanation
# TODO
Explanation: How does matplotlib decides the bins? Actually matplotlib's hist function uses numpy's histogram function under the hood. 
Q: Plot the histogram of movie ratings (IMDB_Rating) using the plt.hist() function.
End of explanation
# TODO: Replace dummy value of below variables with actual values.
n_raw, bins_raw, patches = [1., 2.], [0.5, 1.0], 0.
print(n_raw)
print(bins_raw)
Explanation: Have you noticed that this function returns three objects? Take a look at the documentation here to figure out what they are.
To get the returned three objects:
End of explanation
# TODO: test whether the sum of the numbers in n_raw is equal to the number of movies. 
Explanation: Here, n_raw contains the values of histograms, i.e., the number of movies in each of the 10 bins. Thus, the sum of the elements in n_raw should be equal to the total number of movies. 
Q: Test whether the sum of values in n_raw is equal to the number of movies in the movies dataset
End of explanation
np.diff(bins_raw)
Explanation: The second returned object (bins_raw) is a list containing the edges of the 10 bins: the first bin is [1.4, 2.18], the second [2.18, 2.96], and so on. What's the width of the bins?
End of explanation
min_rating = min(movies['IMDB_Rating'])
max_rating = max(movies['IMDB_Rating'])
print(min_rating, max_rating)
print( (max_rating-min_rating) / 10 )
Explanation: The width is same as the maximum value minus minimum value, divided by 10.
End of explanation
n, bins, patches = plt.hist(movies['IMDB_Rating'], density=True)
print(n)
print(bins)
Explanation: Now, let's plot a normalized (density) histogram.
End of explanation
# TODO
Explanation: The ten bins do not change. But now n represents the density of the data inside each bin. In other words, the sum of the area of each bar will equal to 1. 
Q: Can you verify this?
Hint: the area of each bar is calculated as height * width. You may get something like 0.99999999999999978 instead of 1.
End of explanation
# TODO: If below code gives ValueError with NaN, then there are still missing values in IMDB_Rating and you must remove it.
try:
    np.histogram(movies['IMDB_Rating'])
except ValueError as e:
    print("Resulted in ValueError:", str(e))
plt.hist(movies['IMDB_Rating'])
Explanation: Anyway, these data generated from the hist function is calculated from numpy's histogram function. https://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html 
Note that the result of np.histogram() is same as that of plt.hist().
End of explanation
_ = plt.hist(movies['IMDB_Rating'], bins='auto')
Explanation: If you look at the documentation, you can see that numpy uses simply 10 as the default number of bins. But you can set it manually or set it to be auto, which is the "Maximum of the sturges and fd estimators.". Let's try this auto option.
End of explanation
plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
movies['IMDB_Rating'].hist(bins=3)
plt.subplot(1,2,2)
movies['IMDB_Rating'].hist(bins=20)
Explanation: Consequences of the binning parameter
Let's explore the effect of bin size using small multiples. In matplotlib, you can use subplot to put multiple plots into a single figure. 
For instance, you can do something like:
End of explanation
nbins = [2, 3, 5, 10, 30, 40, 60, 100 ]
figsize = (18, 10)
# TODO
Explanation: What does the argument in plt.subplot(1,2,1) mean? If you're not sure, check out: http://stackoverflow.com/questions/3584805/in-matplotlib-what-does-the-argument-mean-in-fig-add-subplot111
Q: create 8 subplots (2 rows and 4 columns) with the following binsizes.
End of explanation
# TODO: Provide your answer and evidence here
Explanation: Do you see the issues with having too few bins or too many bins? In particular, do you notice weird patterns that emerge from bins=30? 
Q: Can you guess why do you see such patterns? What are the peaks and what are the empty bars? What do they tell you about choosing the binsize in histograms?
End of explanation
N = len(movies)
plt.figure(figsize=(12,4))
# Sqrt 
nbins = int(np.sqrt(N))
plt.subplot(1,3,1)
plt.title("SQRT, {} bins".format(nbins))
movies['IMDB_Rating'].hist(bins=nbins)
# Sturge's formula
nbins = int(np.ceil(np.log2(N) + 1))
plt.subplot(1,3,2)
plt.title("Sturge, {} bins".format(nbins))
movies['IMDB_Rating'].hist(bins=nbins)
# Freedman-Diaconis
# TODO: If below code gives ValueError with NaN, then there are still missing values in IMDB_Rating and you must remove it.
try:
    iqr = np.percentile(movies['IMDB_Rating'], 75) - np.percentile(movies['IMDB_Rating'], 25)
    width = 2*iqr/np.power(N, 1/3)
    nbins = int((max(movies['IMDB_Rating']) - min(movies['IMDB_Rating'])) / width)
    plt.subplot(1,3,3)
    plt.title("F-D, {} bins".format(nbins))
    movies['IMDB_Rating'].hist(bins=nbins)
except ValueError as e:
    print("Resulted in ValueError:", str(e))
Explanation: Formulae for choosing the number of bins.
We can manually choose the number of bins based on those formulae.
End of explanation
plt.figure(figsize=(20,4))
plt.subplot(161)
movies['IMDB_Rating'].hist(bins='fd')
plt.subplot(162)
movies['IMDB_Rating'].hist(bins='doane')
plt.subplot(163)
movies['IMDB_Rating'].hist(bins='scott')
plt.subplot(164)
movies['IMDB_Rating'].hist(bins='rice')
plt.subplot(165)
movies['IMDB_Rating'].hist(bins='sturges')
plt.subplot(166)
movies['IMDB_Rating'].hist(bins='sqrt')
Explanation: But we can also use built-in formulae too. Let's try all of them.
End of explanation
movies['IMDB_Rating'].hist(cumulative=True)
Explanation: Some are decent, but several of them tend to overestimate the good number of bins. As you have more data points, some of the formulae may overestimate the necessary number of bins. Particularly in our case, because of the precision issue, we shouldn't increase the number of bins too much.
Then, how should we choose the number of bins?
So what's the conclusion? use Scott's rule or Sturges' formula? 
No, I think the take-away is that you should understand how the inappropriate number of bins can mislead you and you should try multiple number of bins to obtain the most accurate picture of the data. Although the 'default' may work in most cases, don't blindly trust it! Don't judge the distribution of a dataset based on a single histogram. Try multiple parameters to get the full picture!
CDF (Cumulative distribution function)
Drawing a CDF is easy. Because it's very common data visualization, histogram has an option called cumulative.
End of explanation
movies['IMDB_Rating'].hist(histtype='step', cumulative=True, density=True)
Explanation: You can also combine with options such as histtype and density.
End of explanation
movies['IMDB_Rating'].hist(cumulative=True, density=True, bins=1000)
Explanation: And increase the number of bins.
End of explanation
rating_sorted = movies['IMDB_Rating'].sort_values()
rating_sorted.head()
Explanation: This method works fine. By increasing the number of bins, you can get a CDF in the resolution that you want. But let's also try it manually to better understand what's going on. First, we should sort all the values.
End of explanation
N = len(rating_sorted)
N
Explanation: We need to know the number of data points,
End of explanation
n = 50
np.linspace(1/n, 1.0, num=n)
Explanation: And I think this may be useful for you.
End of explanation
# Implement
Explanation: Q: now you're ready to draw a proper CDF. Draw the CDF plot of this data.
End of explanation
vega_datasets.data.movies.url
# Choose based on your environment
#alt.renderers.enable('notebook')
#alt.renderers.enable('jupyterlab')
#alt.renderers.enable('default')
Explanation: A bit more histogram with altair
As you may remember, you can get a pandas dataframe from vega_datasets package and use it to create visualizations. But, if you use altair, you can simply pass the URL instead of the actual data.
End of explanation
alt.Chart(vega_datasets.data.movies.url).mark_bar().encode(
    alt.X("IMDB_Rating:Q",  bin=True),
    alt.Y('count()')
)
Explanation: As mentioned before, in altair histogram is not special. It is just a plot that use bars (mark_bar()) where X axis is defined by IMDB_Rating with bins (bin=True), and Y axis is defined by count() aggregation function.
End of explanation
alt.Chart(vega_datasets.data.movies.url).mark_bar().encode(
    alt.X('IMDB_Rating', type='quantitative', bin=True),
    alt.Y(aggregate='count', type='quantitative')
)
Explanation: Have you noted that it is IMDB_Rating:Q not IMDB_Rating? This is a shorthand for
End of explanation
from altair import Bin
alt.Chart(vega_datasets.data.movies.url).mark_bar().encode(
    alt.X("IMDB_Rating:Q",  bin=Bin(step=0.09)),
    alt.Y('count()')
)
alt.Chart(vega_datasets.data.movies.url).mark_bar().encode(
    alt.X("IMDB_Rating:Q",  bin=Bin(nice=True, maxbins=20)),
    alt.Y('count()')
)
Explanation: In altair, you want to specify the data types using one of the four categories: quantitative, ordinal, nominal, and temporal. https://altair-viz.github.io/user_guide/encoding.html#data-types
Although you can adjust the bins in altair, it does not encourage you to set the bins directly. For instance, although there is step parameter that directly sets the bin size, there are parameters such as maxbins (maximum number of bins) or minstep (minimum allowable step size), or nice (attemps to make the bin boundaries more human-friendly), that encourage you not to specify the bins directly.
End of explanation
chart1 = alt.Chart(vega_datasets.data.movies.url).mark_bar().encode(
    alt.X("IMDB_Rating:Q",  bin=Bin(step=0.1)),
    alt.Y('count()')
).properties(
    width=300,
    height=150
)
chart2 = alt.Chart(vega_datasets.data.movies.url).mark_bar().encode(
    alt.X("IMDB_Rating:Q",  bin=Bin(nice=True, maxbins=20)),
    alt.Y('count()')
).properties(
    width=300,
    height=150
)
chart1 | chart2
alt.hconcat(chart1, chart2)
Explanation: Composing charts in altair
altair has a very nice way to compose multiple plots. Two histograms side by side? just do the following.
End of explanation
alt.vconcat(chart1, chart2)
chart1 & chart2
Explanation: Vertical commposition?
End of explanation
base = alt.Chart().mark_bar().encode(
    alt.X("IMDB_Rating:Q",  bin=Bin(nice=True, maxbins=20)),
    alt.Y('count()')
).properties(
    width=300,
    height=150
)
chart = alt.vconcat(data=vega_datasets.data.movies.url)
for bin_param in [Bin(step=0.1), Bin(nice=True, maxbins=20)]:
    row = alt.hconcat()
    row |= base.encode(x=alt.X("IMDB_Rating:Q", bin=bin_param), y='count()')
    chart &= row
chart
Explanation: Shall we avoid some repetitions? You can define a base empty chart first and then assign encodings later when you put together multiple charts together. Here is an example: https://altair-viz.github.io/user_guide/compound_charts.html#repeated-charts
Use base chart to produce the chart above:
End of explanation
# TODO
Explanation: Q: Using the base chart approach to create a 2x2 chart where the top row shows the two histograms of IMDB_Rating with maxbins=10 and 50 respectively, and the bottom row shows another two histograms of IMDB_Votes with maxbins=10 and 50.
End of explanation
<END_TASK>