<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Welcome to the 'First steps with pandas'!
After this workshop you can (hopefully) call yourselves Data Scientists!
Gitter
Step1:  What is pandas?
pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.
Why to use it?
It has ready solutions for most of data-related problems
fast development
no reinventing wheel
fewer mistakes/bugs
Step2:  It is easy to pick up
few simple concepts that are very powerful
easy, standardized API
good code readability
It is reasonably fast
Step3:  It has a very cool name.
https
Step4:  DataFrame
DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects.
Creating
Step5:  Meta data
Step6:  Selecting
[string] --&gt; Series
[ list of strings ] --&gt; DataFrame
Step7:  Chaining (most of operations on DataFrame returns new DataFrame or Series)
Step8:  EXERCISE
Create DataFrame presented below in 3 different ways
movie_title  imdb_score
0                                            Avatar          7.9
1          Pirates of the Caribbean
Step9:  With list of dicts
Step10:  With list of Series
Step11:  I/O part I
Reading popular formats / data sources
Step12:  EXERCISE
Load movies from data/movies.csv to variable called movies
Step13:  Analyze what dimensions and columns it has
Step14:  Filtering
Step15:  Boolean indexing
Step16:  Multiple conditions
Step17:  Negation
~ is a negation operator
Step18:  Filtering for containing one of many values (SQL's IN)
Step19:  EXERCISE
What movies has been directed by Clint Eastwood?
Step20:  What movies have earned above $500m?
Step21:  Are there any Polish movies?
Step22:  What are really popular great movies? (> 100k FB likes, > 8.5 IMDB score)
Step23:  In what movies main role was played by brutals like "Jason Statham", "Sylvester Stallone" or god ("Morgan Freeman")?
Step24:  I/O part O
As numpy array
Step25:  As (list) of dicts
Step26:  As popular data format
Step27:  EXERCISE
Create a csv with movie titles and cast (actors) of movies with budget above $200m
Step28:  Create a list of dicts with movie titles and facebook likes of all Christopher Nolan's movies
Step29:  New columns
Step30:  Creating new column
Step31:  Vector operations
Step32:  Map, apply, applymap, str
Step33:  Cheatsheet
map
Step34:  EXERCISE
What are 10 most profitable movies? (ratio between gross and budget)
Step35:  Create a column 'first_genre'. What is the distribution of values in this column?
Step36:  Visualizing data
Step37:  Basic stats
Step38:  Plotting
Step39:  Histogram
Step40:  DataFrames everywhere.. are easy to plot
Step41:  EXERCISE
Create a chart presenting grosses of movies directed by Christopher Nolan
Step42:  What are typical durations of the movies?
Step43:  What is percentage distribution of first genre? (cake)
Step44:  Aggregation
Grouping
https
Step45:  Aggregating
Step46:  EXERCISE
What was maximal gross in each year?
Step47:  Which director earns the most on average?
Step48:  Index related operations
Data alignment on Index
Step49:  Setting index
Step50:  Merging two DFs (a'la SQL join)
Step51:  DateTime operations
Step52:  Filtering
Step53:  Resampling (downsampling and upsampling)
Step54:  EXERCISE
Read Star Wars
Step55:  How many tweets did Star Wars
Step56:  What were the most popular tweeting times of the day for that movie?
Step57:  Advanced topics + Advanced exercises
Filling missing data
Step58:  Scikit-learn
Step59:  More models to try
Step60:  Create a method discovering movies with plot keywords similar to the given list of keywords (i.e. ['magic', 'harry', 'wizard'])
Step61:  Integration with Flask
In the file flask_exercise.py you'll find the scaffolding for Flask app.<br/>Create endpoints returning
<ASSISTANT_TASK:>
Python Code:
import platform
print('Python: ' + platform.python_version())
import numpy as np
print('numpy: ' + np.__version__)
import pandas as pd
print('pandas: ' + pd.__version__)
import scipy
print('scipy: ' + scipy.__version__)
import sklearn
print('scikit-learn: ' + sklearn.__version__)
import matplotlib as plt
print('matplotlib: ' + plt.__version__)
import flask
print('flask: ' + flask.__version__)
Explanation: Welcome to the 'First steps with pandas'!
After this workshop you can (hopefully) call yourselves Data Scientists!
Gitter: https://gitter.im/first-steps-with-pandas-workshop
Before coding, let's check whether we have proper versions of libraries
End of explanation
# In case of no Internet, use:
# pd.read_json('data/cached_Python.json')
(
    pd.read_json('https://raw.githubusercontent.com/Nozdi/first-steps-with-pandas-workshop/master/data/cached_python.json')
    .resample('1W')
    .mean()
    ['daily_views']
)
Explanation: What is pandas?
pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.
Why to use it?
It has ready solutions for most of data-related problems
fast development
no reinventing wheel
fewer mistakes/bugs
End of explanation
some_data = [ list(range(1,100)) for x in range(1,1000) ]
some_df = pd.DataFrame(some_data)
def standard_way(data):
    return [[col*2 for col in row] for row in data]
def pandas_way(df):
    return df * 2
%timeit standard_way(some_data)
%timeit pandas_way(some_df)
Explanation: It is easy to pick up
few simple concepts that are very powerful
easy, standardized API
good code readability
It is reasonably fast
End of explanation
strengths = pd.Series([400, 200, 300, 400, 500])
strengths
names = pd.Series(["Batman", "Robin", "Spiderman", "Robocop", "Terminator"])
names
Explanation: It has a very cool name.
https://c1.staticflickr.com/5/4058/4466498508_35a8172ac1_b.jpg
Library highlights
http://pandas.pydata.org/#library-highlights<br/>
http://pandas.pydata.org/pandas-docs/stable/api.html
Data structures
Series
Series is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.).
End of explanation
heroes = pd.DataFrame({
    'hero': names,
    'strength': strengths
})
heroes
other_heroes = pd.DataFrame([
    dict(hero="Hercules", strength=800),
    dict(hero="Conan")
])
other_heroes
another_heroes = pd.DataFrame([
    pd.Series(["Wonder Woman", 10, 3], index=["hero", "strength", "cookies"]),
    pd.Series(["Xena", 20, 0], index=["hero", "strength", "cookies"])
])
another_heroes
Explanation: DataFrame
DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects.
Creating
End of explanation
another_heroes.columns
another_heroes.shape
another_heroes.info()
Explanation: Meta data
End of explanation
another_heroes['cookies']
another_heroes.cookies
another_heroes[ ['hero', 'cookies'] ]
Explanation: Selecting
[string] --&gt; Series
[ list of strings ] --&gt; DataFrame
End of explanation
another_heroes[['hero', 'cookies']][['cookies']]
another_heroes[['hero', 'cookies']][['cookies']]['cookies']
Explanation: Chaining (most of operations on DataFrame returns new DataFrame or Series)
End of explanation
# Solution here
Explanation: EXERCISE
Create DataFrame presented below in 3 different ways
movie_title  imdb_score
0                                            Avatar          7.9
1          Pirates of the Caribbean: At World's End          7.1
2                                           Spectre          6.8
Help: http://pandas.pydata.org/pandas-docs/stable/dsintro.html#from-dict-of-series-or-dicts
With dict of Series
End of explanation
# Solution here
Explanation: With list of dicts
End of explanation
# Solution here
Explanation: With list of Series
End of explanation
# Uncomment and press tab..
# pd.read_
# SQL, csv, hdf
# pd.read_csv?
# executing bash in jupyter notebook
!head -c 500 data/cached_python.json
pd.read_json('data/cached_python.json')
Explanation: I/O part I
Reading popular formats / data sources
End of explanation
# Solution here
Explanation: EXERCISE
Load movies from data/movies.csv to variable called movies
End of explanation
# Solution here
Explanation: Analyze what dimensions and columns it has
End of explanation
heroes
Explanation: Filtering
End of explanation
heroes['strength'] == 400
heroes[heroes['strength'] == 400]
heroes[heroes['strength'] > 400]
Explanation: Boolean indexing
End of explanation
try:
    heroes[200 < heroes['strength'] < 400]
except ValueError:
    print("This cool Python syntax ain't work :(")
heroes[
    (heroes['strength'] > 200) & 
    (heroes['strength'] < 400)
]
heroes[
    (heroes['strength'] <= 200) |
    (heroes['strength'] >= 400)
]
Explanation: Multiple conditions
End of explanation
~(heroes['strength'] == 400)
heroes['strength'] != 400
heroes[~(
    (heroes['strength'] <= 200) |
    (heroes['strength'] >= 400)
)]
Explanation: Negation
~ is a negation operator
End of explanation
heroes[
    heroes['hero'].isin(['Batman', 'Robin'])
]
Explanation: Filtering for containing one of many values (SQL's IN)
End of explanation
# Solution here
Explanation: EXERCISE
What movies has been directed by Clint Eastwood?
End of explanation
# Solution here
Explanation: What movies have earned above $500m?
End of explanation
# Solution here
Explanation: Are there any Polish movies?
End of explanation
# Solution here
Explanation: What are really popular great movies? (> 100k FB likes, > 8.5 IMDB score)
End of explanation
# Solution here
Explanation: In what movies main role was played by brutals like "Jason Statham", "Sylvester Stallone" or god ("Morgan Freeman")?
End of explanation
heroes.values
Explanation: I/O part O
As numpy array
End of explanation
heroes.to_dict()
heroes.to_dict('records')
Explanation: As (list) of dicts
End of explanation
heroes.to_json()
heroes.to_json(orient='records')
heroes.to_csv()
heroes.to_csv(index=False)
heroes.to_csv('data/heroes.csv', index=False)
Explanation: As popular data format
End of explanation
# Solution here
Explanation: EXERCISE
Create a csv with movie titles and cast (actors) of movies with budget above $200m
End of explanation
# Solution here
Explanation: Create a list of dicts with movie titles and facebook likes of all Christopher Nolan's movies
End of explanation
heroes
Explanation: New columns
End of explanation
heroes['health'] = np.NaN
heroes.head()
heroes['health'] = 100
heroes.head()
heroes['height'] = [180, 170, 175, 190, 185]
heroes
heroes['is_hungry'] = pd.Series([True, False, False, True, True])
heroes
Explanation: Creating new column
End of explanation
heroes['strength'] * 2
heroes['strength'] / heroes['height']
heroes['strength_per_cm'] = heroes['strength'] / heroes['height']
heroes
Explanation: Vector operations
End of explanation
pd.Series([1, 2, 3]).map(lambda x: x**3)
pd.Series(['Batman', 'Robin']).map(lambda x: x[:2])
# however, more idiomatic approach for strings is to do..
pd.Series(['Batman', 'Robin']).str[:2]
pd.Series(['Batman', 'Robin']).str.lower()
pd.Series([
    ['Batman', 'Robin'],
    ['Robocop']
]).map(len)
heroes['code'] = heroes['hero'].map(lambda name: name[:2])
heroes
heroes['effective_strength'] = heroes.apply(
    lambda row: (not row['is_hungry']) * row['strength'],
    axis=1
)
heroes.head()
heroes[['health', 'strength']] = heroes[['health', 'strength']].applymap(
    lambda x: x + 100
)
heroes
Explanation: Map, apply, applymap, str
End of explanation
heroes['strength'].value_counts()
heroes.sort_values('strength')
heroes.sort_values(
    ['is_hungry', 'code'],
    ascending=[False, True]
)
Explanation: Cheatsheet
map: 1 =&gt; 1
apply: n =&gt; 1
applymap: n =&gt; n
Sorting and value counts (bonus skill)
End of explanation
# Solution here
Explanation: EXERCISE
What are 10 most profitable movies? (ratio between gross and budget)
End of explanation
# Solution here
Explanation: Create a column 'first_genre'. What is the distribution of values in this column?
End of explanation
heroes
Explanation: Visualizing data
End of explanation
heroes.describe()
Explanation: Basic stats
End of explanation
%matplotlib inline
pd.Series([1, 2, 3]).plot()
pd.Series([1, 2, 3], index=['Batman', 'Robin', 'Rambo']).plot()
pd.Series([1, 2, 3], index=['Batman', 'Robin', 'Rambo']).plot(kind='bar')
pd.Series([1, 2, 3], index=['Batman', 'Robin', 'Rambo']).plot(
    kind='bar',
    figsize=(15, 6)
)
pd.Series([1, 2, 3], index=['Batman', 'Robin', 'Rambo']).plot(kind='pie')
heroes.plot()
indexed_heroes = heroes.set_index('hero')
indexed_heroes
indexed_heroes.plot()
indexed_heroes.plot(kind='barh')
indexed_heroes.plot(kind='bar', subplots=True, figsize=(15, 15))
indexed_heroes[['height', 'strength']].plot(kind='bar')
heroes.plot(x='hero', y=['height', 'strength'], kind='bar')
# alternative to subplots
heroes.plot(
    x='hero',
    y=['height', 'strength'],
    kind='bar',
    secondary_y='strength',
    figsize=(10,8)
)
heroes.plot(
    x='hero',
    y=['height', 'strength'],
    kind='bar',
    secondary_y='strength',
    title='Super plot of super heroes',
    figsize=(10,8)
)
Explanation: Plotting
End of explanation
heroes.hist(figsize=(10, 10))
heroes.hist(
    figsize=(10, 10),
    bins=2
)
Explanation: Histogram
End of explanation
heroes.describe()['strength'].plot(kind='bar')
Explanation: DataFrames everywhere.. are easy to plot
End of explanation
# Solution here
Explanation: EXERCISE
Create a chart presenting grosses of movies directed by Christopher Nolan
End of explanation
# Solution here
Explanation: What are typical durations of the movies?
End of explanation
# Solution here
Explanation: What is percentage distribution of first genre? (cake)
End of explanation
movie_heroes = pd.DataFrame({
    'hero': ['Batman', 'Robin', 'Spiderman', 'Robocop', 'Lex Luthor', 'Dr Octopus'],
    'movie': ['Batman', 'Batman', 'Spiderman', 'Robocop', 'Spiderman', 'Spiderman'],
    'strength': [400, 100, 400, 560, 89, 300],
    'speed': [100, 10, 200, 1, 20, None],
})
movie_heroes = movie_heroes.set_index('hero')
movie_heroes
movie_heroes.groupby('movie')
list(movie_heroes.groupby('movie'))
Explanation: Aggregation
Grouping
https://www.safaribooksonline.com/library/view/learning-pandas/9781783985128/graphics/5128OS_09_01.jpg
End of explanation
movie_heroes.groupby('movie').size()
movie_heroes.groupby('movie').count()
movie_heroes.groupby('movie')['speed'].sum()
movie_heroes.groupby('movie').mean()
movie_heroes.groupby('movie').apply(
    lambda group: group['strength'] / group['strength'].max()
)
movie_heroes.groupby('movie').agg({
    'speed': 'mean',
    'strength': 'max',
})
movie_heroes = movie_heroes.reset_index()
movie_heroes
movie_heroes.groupby(['movie', 'hero']).mean()
Explanation: Aggregating
End of explanation
# Solution here
Explanation: EXERCISE
What was maximal gross in each year?
End of explanation
# Solution here
Explanation: Which director earns the most on average?
End of explanation
movie_heroes
apetite = pd.DataFrame([
    dict(hero='Spiderman', is_hungry=True),
    dict(hero='Robocop', is_hungry=False)
])
apetite
movie_heroes['is_hungry'] = apetite['is_hungry']
movie_heroes
apetite.index = [2, 3]
movie_heroes['is_hungry'] = apetite['is_hungry']
movie_heroes
Explanation: Index related operations
Data alignment on Index
End of explanation
indexed_movie_heroes = movie_heroes.set_index('hero')
indexed_movie_heroes
indexed_apetite = apetite.set_index('hero')
indexed_apetite
# and alignment works well automagically..
indexed_movie_heroes['is_hungry'] = indexed_apetite['is_hungry']
indexed_movie_heroes
Explanation: Setting index
End of explanation
movie_heroes
apetite
# couple of other arguments available here
pd.merge(
    movie_heroes[['hero', 'speed']],
    apetite,
    on=['hero'],
    how='outer'
)
Explanation: Merging two DFs (a'la SQL join)
End of explanation
spiderman_meals = pd.DataFrame([
        dict(time='2016-10-15 10:00', calories=300),
        dict(time='2016-10-15 13:00', calories=900),
        dict(time='2016-10-15 15:00', calories=1200),
        dict(time='2016-10-15 21:00', calories=700),
        dict(time='2016-10-16 07:00', calories=1600),
        dict(time='2016-10-16 13:00', calories=600),
        dict(time='2016-10-16 16:00', calories=900),
        dict(time='2016-10-16 20:00', calories=500),
        dict(time='2016-10-16 21:00', calories=300),
        dict(time='2016-10-17 08:00', calories=900),
    ])
spiderman_meals
spiderman_meals.dtypes
spiderman_meals['time'] = pd.to_datetime(spiderman_meals['time'])
spiderman_meals.dtypes
spiderman_meals
spiderman_meals = spiderman_meals.set_index('time')
spiderman_meals
spiderman_meals.index
Explanation: DateTime operations
End of explanation
spiderman_meals["2016-10-15"]
spiderman_meals["2016-10-16 10:00":]
spiderman_meals["2016-10-16 10:00":"2016-10-16 20:00"]
spiderman_meals["2016-10"]
Explanation: Filtering
End of explanation
spiderman_meals.resample('1D').sum()
spiderman_meals.resample('1H').mean()
spiderman_meals.resample('1H').ffill()
spiderman_meals.resample('1D').first()
Explanation: Resampling (downsampling and upsampling)
End of explanation
# Solution here
Explanation: EXERCISE
Read Star Wars: The Force Awakens's tweets from data/theforceawakens_tweets.csv. Create DateTimeIndex from created_at column.
End of explanation
# Solution here
Explanation: How many tweets did Star Wars: The Force Awakens have in each of last days?
End of explanation
# Solution here
Explanation: What were the most popular tweeting times of the day for that movie?
End of explanation
heroes_with_missing = pd.DataFrame([
        ('Batman', None, None),
        ('Robin', None, 100),
        ('Spiderman', 400, 90),
        ('Robocop', 500, 95),
        ('Terminator', 600, None)
    ], columns=['hero', 'strength', 'health'])
heroes_with_missing
heroes_with_missing.dropna()
heroes_with_missing.fillna(0)
heroes_with_missing.fillna({'strength': 10, 'health': 20})
heroes_with_missing.fillna(heroes_with_missing.min())
heroes_with_missing.fillna(heroes_with_missing.median())
Explanation: Advanced topics + Advanced exercises
Filling missing data
End of explanation
pd.DataFrame({'x': [1, 2], 'y': [10, 20]}).plot(x='x', y='y', kind='scatter')
from sklearn.linear_model import LinearRegression
X=[ [1], [2] ]
y=[ 10, 20 ]
clf = LinearRegression()
clf.fit(X, y)
clf.predict([ [0.5], [2], [4] ])
X = np.array([ [1], [2] ])
y = np.array([ 10, 20 ])
X
clf = LinearRegression()
clf.fit(X, y)
clf.predict( np.array([ [0.5], [2], [4] ]) )
train_df = pd.DataFrame([
        (1, 10),
        (2, 20),
    ], columns=['x', 'y'])
train_df
clf = LinearRegression()
clf.fit(train_df[['x']], train_df['y'])
clf.predict([[0.5]])
test_df = pd.DataFrame({'x': [0.5, 1.5, 4]})
test_df
clf.predict(test_df[['x']])
test_df['y'] = clf.predict(test_df[['x']])
test_df
train_df['color'] = 'blue'
test_df['color'] = 'red'
all_df = train_df.append(test_df)
all_df.plot(x='x', y='y', kind='scatter', figsize=(10, 8), color=all_df['color'])
Explanation: Scikit-learn
End of explanation
# Solution here
Explanation: More models to try: http://scikit-learn.org/stable/supervised_learning.html#supervised-learning
EXERCISE
Integration with scikit-learn: Create a model that tries to predict gross of movie. Use any features of the movies dataset.
End of explanation
# Solution here
Explanation: Create a method discovering movies with plot keywords similar to the given list of keywords (i.e. ['magic', 'harry', 'wizard'])
End of explanation
# Solution in flask_exercise.py
Explanation: Integration with Flask
In the file flask_exercise.py you'll find the scaffolding for Flask app.<br/>Create endpoints returning:
  - all movie titles available in the movies dataset
  - 10 worst rated movies ever
  - 10 best rated (imdb_score) movies in a given year
End of explanation
<END_TASK>