<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 W3 Lab Assignment
Submit the .ipynb file to Canvas with file name w03_lab_lastname_firstname.ipynb.
In this lab, we will introduce pandas, matplotlib, and seaborn and continue to use the imdb.csv file from the last lab.
There will be some exercises, and as usual, write your code in the empty cells to answer them.
Importing libraries
I think some of you have already used pandas. Pandas is a library for high-performance data analysis, and makes makes tedious jobs of reading, manipulating, analyzing data super easy and nice. You can even plot directly using pandas. If you used R before, you'll see a lot of similarity with the R's dataframe and pandas's dataframe.
Step1:  Matplotlib magic
Jupyter notebook provides several magic commands. These are the commands that you can use only in the notebook (not in IDLE for instance). One of the greatest magic command is matplotlib inline, which displays plots within the notebook instead of creating figure files.
Step2:  There are many ways to import matplotlib, but the most common way is
Step3:  Q1
Step4:  Let's look at the first few rows to get some sense of the data.
Step5:  You can see more, or less lines of course
Step6:  You can extract one column by using dictionary-like expression
Step7:  or select multiple columns
Step8:  To get the first 10 rows
Step9:  We can also select both rows and columns. For example, to select the first 10 rows of the 'Year' and 'Rating' columns
Step10:  You can swap the order of rows and columns. 
But, when you deal with large datasets, You may want to stick to this principle
Step11:  It is very easy to answer the question of the number of movies per year. The value_counts() function counts how many times each data value (year) appears.
Step12:  To calculate average ratings and votes
Step13:  or you can even do
Step14:  To get the median ratings of movies in 1990s, we first select only movies in that decade
Step15:  Then, we can do the calculation
Step16:  Finally, if we want to know the top 10 movies in 1990s, we can use the sort() function
Step17:  Exercise
Calculate the following basic characteristics of ratings of movies only in 1994
Step18:  Q2
Step19:  Exercise
Can you plot the histogram of ratings of the movies between 2000 and 2014?
Step20:  Q3
Step21:  Exercise
Let's try to make some style changes to the plot
Step22:  Q4
Step23:  Let's do nothing and just run the histgram again
Step24:  We can use the distplot() function to plot the histogram.
Step25:  Exercise
Read the document about the function and make the following changes
<ASSISTANT_TASK:>
Python Code:
import pandas as pd  
import numpy as np
import warnings
warnings.filterwarnings('ignore')
Explanation: W3 Lab Assignment
Submit the .ipynb file to Canvas with file name w03_lab_lastname_firstname.ipynb.
In this lab, we will introduce pandas, matplotlib, and seaborn and continue to use the imdb.csv file from the last lab.
There will be some exercises, and as usual, write your code in the empty cells to answer them.
Importing libraries
I think some of you have already used pandas. Pandas is a library for high-performance data analysis, and makes makes tedious jobs of reading, manipulating, analyzing data super easy and nice. You can even plot directly using pandas. If you used R before, you'll see a lot of similarity with the R's dataframe and pandas's dataframe.
End of explanation
%matplotlib inline 
Explanation: Matplotlib magic
Jupyter notebook provides several magic commands. These are the commands that you can use only in the notebook (not in IDLE for instance). One of the greatest magic command is matplotlib inline, which displays plots within the notebook instead of creating figure files.
End of explanation
import matplotlib.pyplot as plt 
Explanation: There are many ways to import matplotlib, but the most common way is:
End of explanation
df = pd.read_csv('imdb.csv', delimiter='\t')
Explanation: Q1: Revisting W2 lab
Let's revisit last week's exercise with pandas. It's very easy to read CSV files with pandas, using the panda.read_csv() function. This function has many many options and it may be worthwhile to take a look at available options. Things that you need to be careful are: 
delimiter or sep: the data file may use ',', tab, or any weird character to separate fields. You can't read data properly if this option is incorrect. 
header: some data files have "header" row that contains the names of the columns. If you read it as data or use the first row as the header, you'll have problems. 
na_values or na_filter: often the dataset is incomplete and contains missing data (NA, NaN (not a number), etc.). It's very important to take care of them properly. 
You don't need to create dictionaries and other data structures. Pandas just imports the whole table into a data structure called DataFrame. You can do all kinds of interesting manipulation with the DataFrame.
End of explanation
df.head()
Explanation: Let's look at the first few rows to get some sense of the data.
End of explanation
df.head(2)
Explanation: You can see more, or less lines of course
End of explanation
df['Year'].head(3)
Explanation: You can extract one column by using dictionary-like expression
End of explanation
df[['Year','Rating']].head(3)
Explanation: or select multiple columns
End of explanation
df[:10]
Explanation: To get the first 10 rows
End of explanation
df[['Year','Rating']][:10]
Explanation: We can also select both rows and columns. For example, to select the first 10 rows of the 'Year' and 'Rating' columns:
End of explanation
df[:10][['Year','Rating']]
Explanation: You can swap the order of rows and columns. 
But, when you deal with large datasets, You may want to stick to this principle: 
Try to reduce the size of the dataset you are handling as soon as possible, and as much as possible. 
For instance, if you have a billion rows with three columns, getting the small row slice (df[:10]) and working with this small slice can be much better than getting the column slice (df['Year']) and working with this slice (still contains billion items).
End of explanation
print( min(df['Year']), df['Year'].min(), max(df['Year']), df['Year'].max() )
year_nummovies = df["Year"].value_counts()
year_nummovies.head()
Explanation: It is very easy to answer the question of the number of movies per year. The value_counts() function counts how many times each data value (year) appears.
End of explanation
print( np.mean(df['Rating']), np.mean(df['Votes']) )
Explanation: To calculate average ratings and votes
End of explanation
print( df['Rating'].mean() )
Explanation: or you can even do
End of explanation
geq = df['Year'] >= 1990
leq = df['Year'] <= 1999
movie_nineties = df[geq & leq]
movie_nineties.head()
Explanation: To get the median ratings of movies in 1990s, we first select only movies in that decade
End of explanation
print( movie_nineties['Rating'].median(), movie_nineties['Votes'].median() )
Explanation: Then, we can do the calculation
End of explanation
sorted_by_rating = movie_nineties.sort('Rating', ascending=False)
sorted_by_rating[:10]
Explanation: Finally, if we want to know the top 10 movies in 1990s, we can use the sort() function:
End of explanation
# implement here
df[(df['Year']==1994)]['Rating'].describe([.1,0.9])
df['Rating'].median()
Explanation: Exercise
Calculate the following basic characteristics of ratings of movies only in 1994: 10th percentile, median, mean, 90th percentile.
http://docs.scipy.org/doc/numpy/reference/generated/numpy.percentile.html
http://pandas.pydata.org/pandas-docs/stable/text.html
Write your code in the cell below
End of explanation
df['Year'].hist()
Explanation: Q2: Basic plotting with pandas
Pandas provides some easy ways to draw plots by using matplotlib. Dataframe object has several plotting functions. For instance,
End of explanation
# implement here
plt.hist(df[(df['Year']>2000) & (df['Year']<2014)]['Rating'],bins = 10)
Explanation: Exercise
Can you plot the histogram of ratings of the movies between 2000 and 2014?
End of explanation
plt.hist(df['Rating'], bins=10)
Explanation: Q3: Basic plotting with matplotlib
Let's plot the histogram of ratings using the pyplot.hist() function.
End of explanation
# implement here
plt.hist(df[(df['Year']>2000) & (df['Year']<2014)]['Rating'],bins = 20,facecolor='g')
plt.xlabel('bins')
plt.ylabel('# Ratings')
plt.title('Histogram of Rating Distribution for years 2000-2014')
plt.grid(True)
Explanation: Exercise
Let's try to make some style changes to the plot:
change the color from blue to whatever you want
http://matplotlib.org/users/pyplot_tutorial.html#working-with-text
http://matplotlib.org/api/colors_api.html
add labels of x and y axis
change the number of bins to 20
End of explanation
import seaborn as sns
Explanation: Q4: Basic plotting with Seaborn
Seaborn sits on the top of matplotlib and makes it easier to draw statistical plots. Most plots that you create with Seaborn can be created with matplotlib. It just typically requires a lot more work. 
Be sure seaborn has been installed on your computer, otherwise run
conda install seaborn
End of explanation
plt.hist(df['Rating'], bins=10)
Explanation: Let's do nothing and just run the histgram again
End of explanation
sns.distplot(df['Rating'])
Explanation: We can use the distplot() function to plot the histogram.
End of explanation
# implement here
sns.distplot(df['Rating'],bins = 10,kde=False)
plt.xlabel('bins')
plt.ylabel('# Ratings')
plt.title('Histogram of Rating Distribution for years 2000-2014')
Explanation: Exercise
Read the document about the function and make the following changes: http://stanford.edu/~mwaskom/software/seaborn/generated/seaborn.distplot.html
change the number of bins to 10;
not to show kde;
End of explanation
<END_TASK>