<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Groupby operations
Some imports
Step1:  Recap
Step2:  Using the filtering and reductions operations we have seen in the previous notebooks, we could do something like
Step3:  Pandas does not only let you group by a column name. In df.groupby(grouper) can be many things
Step4:  And now applying this on some real data
These exercises are based on the PyCon tutorial of Brandon Rhodes (so all credit to him!) and the datasets he prepared for that. You can download these data from here
Step5:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step6:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step7:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step8:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step9:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step10:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step11:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step12:  Transforms
Sometimes you don't want to aggregate the groups, but transform the values in each group. This can be achieved with transform
Step13:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step14:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step15:  Intermezzo
Step16:  In pandas, those methods (together with some additional methods) are also available for string Series through the .str accessor
Step17:  For an overview of all string methods, see
Step18:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step19:  Value counts
A useful shortcut to calculate the number of occurences of certain values is value_counts (this is somewhat equivalent to df.groupby(key).size()))
For example, what are the most occuring movie titles?
Step20:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step21:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step22:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step23:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step24:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step25:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step26:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step27:  <div class="alert alert-success">
    <b>EXERCISE</b>
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
try:
    import seaborn
except ImportError:
    pass
pd.options.display.max_rows = 10
Explanation: Groupby operations
Some imports:
End of explanation
df = pd.DataFrame({'key':['A','B','C','A','B','C','A','B','C'],
                   'data': [0, 5, 10, 5, 10, 15, 10, 15, 20]})
df
Explanation: Recap: the groupby operation (split-apply-combine)
The "group by" concept: we want to apply the same function on subsets of your dataframe, based on some key to split the dataframe in subsets
This operation is also referred to as the "split-apply-combine" operation, involving the following steps:
Splitting the data into groups based on some criteria
Applying a function to each group independently
Combining the results into a data structure
<img src="img/splitApplyCombine.png">
Similar to SQL GROUP BY
The example of the image in pandas syntax:
End of explanation
df.groupby('key').aggregate('sum')  # np.sum
df.groupby('key').sum()
Explanation: Using the filtering and reductions operations we have seen in the previous notebooks, we could do something like:
df[df['key'] == "A"].sum()
df[df['key'] == "B"].sum()
...
But pandas provides the groupby method to do this:
End of explanation
df.groupby(lambda x: x % 2).mean()
Explanation: Pandas does not only let you group by a column name. In df.groupby(grouper) can be many things:
Series (or string indicating a column in df)
function (to be applied on the index)
dict : groups by values
levels=[], names of levels in a MultiIndex
End of explanation
cast = pd.read_csv('data/cast.csv')
cast.head()
titles = pd.read_csv('data/titles.csv')
titles.head()
Explanation: And now applying this on some real data
These exercises are based on the PyCon tutorial of Brandon Rhodes (so all credit to him!) and the datasets he prepared for that. You can download these data from here: titles.csv and cast.csv and put them in the /data folder.
cast dataset: different roles played by actors/actresses in films
title: title of the film
name: name of the actor/actress
type: actor/actress
n: the order of the role (n=1: leading role)
End of explanation
# %load snippets/04b - Advanced groupby operations8.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Using groupby(), plot the number of films that have been released each decade in the history of cinema.
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations9.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Use groupby() to plot the number of "Hamlet" films made each decade.
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations10.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: How many leading (n=1) roles were available to actors, and how many to actresses, in each year of the 1950s?
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations11.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: List the 10 actors/actresses that have the most leading roles (n=1) since the 1990's.
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations12.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Use groupby() to determine how many roles are listed for each of The Pink Panther movies.
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations13.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: List, in order by year, each of the films in which Frank Oz has played more than 1 role.
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations15.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: List each of the characters that Frank Oz has portrayed at least twice.
</div>
End of explanation
df
df.groupby('key').transform('mean')
def normalize(group):
    return (group - group.mean()) / group.std()
df.groupby('key').transform(normalize)
df.groupby('key').transform('sum')
Explanation: Transforms
Sometimes you don't want to aggregate the groups, but transform the values in each group. This can be achieved with transform:
End of explanation
# %load snippets/04b - Advanced groupby operations21.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Add a column to the `cast` dataframe that indicates the number of roles for the film.
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations22.py
# %load snippets/04b - Advanced groupby operations23.py
# %load snippets/04b - Advanced groupby operations24.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Calculate the ratio of leading actor and actress roles to the total number of leading roles per decade.     
</div>
Tip: you can to do a groupby twice in two steps,  once calculating the numbers, and then the ratios.
End of explanation
s = 'Bradwurst'
s.startswith('B')
Explanation: Intermezzo: string manipulations
Python strings have a lot of useful methods available to manipulate or check the content of the string:
End of explanation
s = pd.Series(['Bradwurst', 'Kartoffelsalat', 'Sauerkraut'])
s.str.startswith('B')
Explanation: In pandas, those methods (together with some additional methods) are also available for string Series through the .str accessor:
End of explanation
# %load snippets/04b - Advanced groupby operations29.py
# %load snippets/04b - Advanced groupby operations30.py
Explanation: For an overview of all string methods, see: http://pandas.pydata.org/pandas-docs/stable/api.html#string-handling
<div class="alert alert-success">
    <b>EXERCISE</b>: We already plotted the number of 'Hamlet' films released each decade, but not all titles are exactly called 'Hamlet'. Give an overview of the titles that contain 'Hamlet', and that start with 'Hamlet':
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations31.py
# %load snippets/04b - Advanced groupby operations32.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: List the 10 movie titles with the longest name.
</div>
End of explanation
titles.title.value_counts().head()
Explanation: Value counts
A useful shortcut to calculate the number of occurences of certain values is value_counts (this is somewhat equivalent to df.groupby(key).size()))
For example, what are the most occuring movie titles?
End of explanation
# %load snippets/04b - Advanced groupby operations34.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Which years saw the most films released?
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations35.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Plot the number of released films over time
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations36.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Plot the number of "Hamlet" films made each decade.
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations37.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: What are the 11 most common character names in movie history?
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations38.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Which actors or actresses appeared in the most movies in the year 2010?
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations39.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Plot how many roles Brad Pitt has played in each year of his career.
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations40.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: What are the 10 most film titles roles that start with the word "The Life"?
</div>
End of explanation
# %load snippets/04b - Advanced groupby operations41.py
# %load snippets/04b - Advanced groupby operations42.py
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: How many leading (n=1) roles were available to actors, and how many to actresses, in the 1950s? And in 2000s?
</div>
End of explanation
<END_TASK>