<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Practical Optimisations for Pandas 🐼
Eyal Trabelsi
About Me 🙈
Software Engineer at Salesforce 👷
Big passion for python, data and performance optimisations 🐍🤖
Online at medium | twitter 🌐
What's Pandas?🐼
Library for data manipulation
Dataset on Memory
Widely used 
Performance?! Why ?🤨
Fast is better than slow 🐇
Memory efficiency is good 💾
Saving money is awesome 💸
Hardware will only take you so far 💻
Ok now that i have got you attention, the next question i want to tackle is when should we optimize our code
When ⏰
since program readability is our top priority, as python aim to make the programmer life easier, we should only optimize our code when needed.
so  in other words All optimization are premature.
unless
Program doesn't meet requirements 🚔
wheter its too slow for the user
or whether its take too much memory
Program execution affects development pace 👷
Only bottlenecks
We should only optimize the program bottlenecks
whether memory and execution time in order to be productive
There is a great talk about how it can be done using profiling and the link is provieded
Well Tested
Like every refactoring task, you want to have same behaviour and return the same restult.
The best way to feel safe while refactoring is  by writting tests
altough people mostly think the problem lays in python realms its not actually the case and i will show it to you today
so hopefully now that i have gained your trust 
we can tackle  The one milion dollar question.
how can we optimize our pandas code.
Dataset 📉
Step1:  Our dataset is a row 1m data that contain meal invoices
Before i begin its important to state that all these optimization depends on the size of your data and for small dataset these optimizations might be irelevant
How 👀
Use What You Need 🧑
Keep needed columns only
For many use cases you dont actually need the entire dataset
There might be unneeded columns which we dont use in our analysis or data manipulation and this can save a lot of memory
Keep needed rows only
again For many usecase you dont need all the rows.
For example if you want to understand the dataset layout you dont need all 1m rows
this will reduce both your memory footprint and execution time
Dont Reinvent the Wheel 🎡
Vast ecosystem
Use existing solutions
Less bugs
Highly optimized
For example instead of implmenting kmeans by your own
use scipy/sckitlearn implementation, not only you will have less bugs
but it will probably be highly optimized
Avoid Loops ♾
Pandas is design for vector manipulations Which make loops inefficient
a Rookie mistake in pandas will be to just loop over all the rows" either using iterows or regular iterations
Bad Option 😈
Step2:  As you might guessed this bring us  unsatisfied results of half an hour.
there is a much Nicer way and  it is to use map or apply.
Better Option 🤵
apply accepts any user defined function that applies a transformation/aggregation on a DataFrame.
Step3:  As we can the performance look better as it took 21 seconds and about 100 times faster then iterows 
but can we do any better? The obvious answer is yes using vectorization
100x Improvement In Execution Time ⌛
Iterrow is evil 😈
Best Option 👼
Step4:  Vectorization is the process of executing operations on entire arrays.
Pandas/numpy/scipy includes a generous collection of vectorized functions from mathematical operations to aggregations.
Step5:  We can see the benefit of vetorized function right away we got from  to 2ms and about 100,00.
Another 8000x Improvement In Execution Time ⌛
Use Vectorized Operations 😇
Picking the Right Type 🌈
Motivation 🏆
Step6:  Where We Stand 🌈
Step7:  Dataframe Size 🏋️
Step8:  Columns Sizes 🏋️
Step9:  I hope its clear that the type of the column effect the memory foot print.
There is a great picture explain
Supported Types 🌈
int64 
float64
bool
objects
datetime64
timedelta
Category
The category type is good when the same elements occur over and over again.
and its New in version 0.23.0.
Sparse Types
The sparse types is good when most of the array includes nulls.
and its New in version 0.24.0
Nullable Integer/Nullable Bolean
The Nullable type is good when element are integer/boolean and includes nulls.
This is because NaN is a float and it forces the entire array to be cast as float and thus has a bigger memory footprint. 
and its New in version 0.24.0
so its pretty obvious that we should aim for the type that has lowest memory footprint with the same functionality
Optimizing Types 🌈
Loading dataframes with specific types
Use to_numeric/to_datetime/to_timedelta functions with downcast parameter
Step10:  Optimized Types 🌈
Dataframe Size 🏋️
Step11:  Columns Sizes 🏋️
Step12:  12x Improvement In Memory ⌛
its important to note that i have loaded the dataframe with very naive types in order to emphasis the change
but in most use cases you could expect to at least halve the memory foot print
Types Magic Explanation 🧙
DataFrame is stored in seperated blocks depending on thier types.
Under the hood, pandas groups the columns into blocks of values of the same type.
For blocks representing numeric values like integers and floats, pandas combines the columns and stores them as a NumPy ndarray. Also for extension arrays.
The NumPy ndarray is built around a C array, and the values are stored in a contiguous block of memory. Due to this storage scheme, accessing a slice of values is incredibly fast.
Custom Types 🦸🏼🦸‍♀️
This optimization is for extreme cases as it requires a lot of effort and skills.
basicly its implementing you own custom types using extension array
Your Own Types
Open Sourced Types like cyberpandas and geopandas 
There are open sourced Types like cyberpandas for ip like objects and geopadnas for spatial like objects
Pandas Usage 🐼
Chunks 🍰
Splitting large data to smaller parts
Work with large datasets
Step13:  Mean/Sum/Mode/Min/etc optimization 🧮
Types matter
Step14:  20x Performance Improvement⌛
DataFrame Serialization 🏋
Various file formats
Loading time
Saving time
Disk space
Query/Eval 🧬
Use numexpr if installed
Improve Execution Time 👍
The expected behaviour is up to 2 time faster
Improve Memory 👍
The main benefit of eval/query is the saved memory
NumPy allocate memory to every itermediate step.
and this Gives you the ability to compute this type of compound expression element by element, without the need to allocate full intermediate arrays.
Not all Operations are supported 👎
Example
Step15:  % 20 Performance Improvement⌛
Use on big datasets 🧞
The pandas API indicate to only use eval on  more than 10,000 rows, as the traditional method is faster for smaller arrays.
Concat vs Append ➕
Every append creates a new dataframe object
Multiple appends are inefficient
Use concat
Sorting Optimization 📟
Vanilla python
Pandas 
Numpy
Pytorch/Tensorflow
GroupBy Optimizations 👩‍👩‍👧
Filter early
Custom functions are slow
Extract logic for custom functions when possible
Merge Optimization 🔍
Filter/Aggregate early
Semi join
Join on index
Compiled Code ?! Why 🤯
Python dynamic nature 
No compilation optimization
Pure Python can be slow
Step16:  Cython
the Cython project which converts Python code to compatiable C code
Numba
and the second one the Numba project, which converts Python code to fast LLVM bytecode
Step17:  Cython and Numba for the rescue 👨‍🚒
Cython 🤯
Up to 50x speedup from pure python 👍
Learning Curve 👎
Separated Compilation Step 👎 👍
The compilation is both a con and a pro
on one hand, there is an additional work to integrate the cython code 
and on the other hand, there is no compilation overhead on runtime
Step18:  Example
Step19:  49x  Performance Improvement⌛
Numba 🤯
Up to 200x speedup from pure python  👍
Easy 👍
using numba is really easy its simply adding a decorator to a method
Highly Configurable  👍
Debugging 👎 👍
Debugging is both a con and a pro
on one hand, debugging the buissness logic is easy we can just remove the decorator and then we debug it as python code
on the other hand, if there is an issue in the compiled code itself its not quite easy to debug these issues.
Mostly Numeric 👎 
Example
Step20:  43x  Performance Improvement⌛
Step21:  1️⃣ Vectorized methods
2️⃣ Numba
3️⃣ Cython
General Python Optimizations 🐍
Since pandas is writen in python we can optimize our pandas code by using python optimization 
Caching 🏎
Avoid unnecessary work/computation.
Faster code 
Intermediate Variables👩‍👩‍👧‍👧
Intermediate calculations 
Memory foot print of both objects
Smarter variables allocation
Step22:  Example
<ASSISTANT_TASK:>
Python Code:
! pip install numba numexpr
import math
import time
import warnings
from dateutil.parser import parse
import janitor
import numpy as np
import pandas as pd
from numba import jit
from sklearn import datasets
from pandas.api.types import is_datetime64_any_dtype as is_datetime
warnings.filterwarnings("ignore", category=pd.errors.DtypeWarning)
pd.options.display.max_columns = 999
path = 'https://raw.githubusercontent.com/FBosler/you-datascientist/master/invoices.csv'
def load_dataset(naivly=False):
    df = (pd.concat([pd.read_csv(path)
                       .clean_names()
                       .remove_columns(["meal_id", "company_id"]) 
                     for i in range(20)])
            .assign(meal_tip=lambda x: x.meal_price.map(lambda x: x * 0.2))
            .astype({"meal_price": int})
            .rename(columns={"meal_price": "meal_price_with_tip"}))
    if naivly:
        for col in df.columns:
            df[col] = df[col].astype(object)
    return df
df = load_dataset()
df.head()
Explanation: Practical Optimisations for Pandas 🐼
Eyal Trabelsi
About Me 🙈
Software Engineer at Salesforce 👷
Big passion for python, data and performance optimisations 🐍🤖
Online at medium | twitter 🌐
What's Pandas?🐼
Library for data manipulation
Dataset on Memory
Widely used 
Performance?! Why ?🤨
Fast is better than slow 🐇
Memory efficiency is good 💾
Saving money is awesome 💸
Hardware will only take you so far 💻
Ok now that i have got you attention, the next question i want to tackle is when should we optimize our code
When ⏰
since program readability is our top priority, as python aim to make the programmer life easier, we should only optimize our code when needed.
so  in other words All optimization are premature.
unless
Program doesn't meet requirements 🚔
wheter its too slow for the user
or whether its take too much memory
Program execution affects development pace 👷
Only bottlenecks
We should only optimize the program bottlenecks
whether memory and execution time in order to be productive
There is a great talk about how it can be done using profiling and the link is provieded
Well Tested
Like every refactoring task, you want to have same behaviour and return the same restult.
The best way to feel safe while refactoring is  by writting tests
altough people mostly think the problem lays in python realms its not actually the case and i will show it to you today
so hopefully now that i have gained your trust 
we can tackle  The one milion dollar question.
how can we optimize our pandas code.
Dataset 📉
End of explanation
def iterrows_original_meal_price(df):
    for i, row in df.iterrows():
        df.loc[i]["original_meal_price"] = row["meal_price_with_tip"] - row["meal_tip"]
    return df
%%timeit -r 1 -n 1
iterrows_original_meal_price(df)
Explanation: Our dataset is a row 1m data that contain meal invoices
Before i begin its important to state that all these optimization depends on the size of your data and for small dataset these optimizations might be irelevant
How 👀
Use What You Need 🧑
Keep needed columns only
For many use cases you dont actually need the entire dataset
There might be unneeded columns which we dont use in our analysis or data manipulation and this can save a lot of memory
Keep needed rows only
again For many usecase you dont need all the rows.
For example if you want to understand the dataset layout you dont need all 1m rows
this will reduce both your memory footprint and execution time
Dont Reinvent the Wheel 🎡
Vast ecosystem
Use existing solutions
Less bugs
Highly optimized
For example instead of implmenting kmeans by your own
use scipy/sckitlearn implementation, not only you will have less bugs
but it will probably be highly optimized
Avoid Loops ♾
Pandas is design for vector manipulations Which make loops inefficient
a Rookie mistake in pandas will be to just loop over all the rows" either using iterows or regular iterations
Bad Option 😈
End of explanation
def apply_original_meal_price(df):
    df["original_meal_price"] = df.apply(lambda x: x['meal_price_with_tip'] - x['meal_tip'], axis=1)
    return df
%%timeit 
apply_original_meal_price(df)
Explanation: As you might guessed this bring us  unsatisfied results of half an hour.
there is a much Nicer way and  it is to use map or apply.
Better Option 🤵
apply accepts any user defined function that applies a transformation/aggregation on a DataFrame.
End of explanation
def vectorized_original_meal_price(df):
    df["original_meal_price"] = df["meal_price_with_tip"] - df["meal_tip"] 
    return df
Explanation: As we can the performance look better as it took 21 seconds and about 100 times faster then iterows 
but can we do any better? The obvious answer is yes using vectorization
100x Improvement In Execution Time ⌛
Iterrow is evil 😈
Best Option 👼
End of explanation
%%timeit 
vectorized_original_meal_price(df)
Explanation: Vectorization is the process of executing operations on entire arrays.
Pandas/numpy/scipy includes a generous collection of vectorized functions from mathematical operations to aggregations.
End of explanation
ones = np.ones(shape=5000)
ones
types = ['object', 'complex128', 'float64', 'int64', 'int32', 'int16', 'int8', 'bool']
df = pd.DataFrame(dict([(t, ones.astype(t)) for t in types]))
df.memory_usage(index=False, deep=True)
Explanation: We can see the benefit of vetorized function right away we got from  to 2ms and about 100,00.
Another 8000x Improvement In Execution Time ⌛
Use Vectorized Operations 😇
Picking the Right Type 🌈
Motivation 🏆
End of explanation
df = load_dataset(naivly=True)
Explanation: Where We Stand 🌈
End of explanation
df.memory_usage(deep=True).sum()
Explanation: Dataframe Size 🏋️
End of explanation
df.memory_usage(deep=True)
df.dtypes
Explanation: Columns Sizes 🏋️
End of explanation
df = df.astype({'order_id': 'category',
                'date': 'category',
                'date_of_meal': 'category',
                'participants': 'category',
                'meal_price': 'int16',
                'type_of_meal': 'category',
                'heroes_adjustment': 'bool',
                'meal_tip': 'float32'})
Explanation: I hope its clear that the type of the column effect the memory foot print.
There is a great picture explain
Supported Types 🌈
int64 
float64
bool
objects
datetime64
timedelta
Category
The category type is good when the same elements occur over and over again.
and its New in version 0.23.0.
Sparse Types
The sparse types is good when most of the array includes nulls.
and its New in version 0.24.0
Nullable Integer/Nullable Bolean
The Nullable type is good when element are integer/boolean and includes nulls.
This is because NaN is a float and it forces the entire array to be cast as float and thus has a bigger memory footprint. 
and its New in version 0.24.0
so its pretty obvious that we should aim for the type that has lowest memory footprint with the same functionality
Optimizing Types 🌈
Loading dataframes with specific types
Use to_numeric/to_datetime/to_timedelta functions with downcast parameter
End of explanation
df.memory_usage(deep=True).sum()
Explanation: Optimized Types 🌈
Dataframe Size 🏋️
End of explanation
df.memory_usage(deep=True)
df.dtypes
Explanation: Columns Sizes 🏋️
End of explanation
def proccess():
    pass
def proccess_file(huge_file_path, chunksize = 10 ** 6):
    for chunk in pd.read_csv(path, chunksize=chunksize):
        process(chunk)    
Explanation: 12x Improvement In Memory ⌛
its important to note that i have loaded the dataframe with very naive types in order to emphasis the change
but in most use cases you could expect to at least halve the memory foot print
Types Magic Explanation 🧙
DataFrame is stored in seperated blocks depending on thier types.
Under the hood, pandas groups the columns into blocks of values of the same type.
For blocks representing numeric values like integers and floats, pandas combines the columns and stores them as a NumPy ndarray. Also for extension arrays.
The NumPy ndarray is built around a C array, and the values are stored in a contiguous block of memory. Due to this storage scheme, accessing a slice of values is incredibly fast.
Custom Types 🦸🏼🦸‍♀️
This optimization is for extreme cases as it requires a lot of effort and skills.
basicly its implementing you own custom types using extension array
Your Own Types
Open Sourced Types like cyberpandas and geopandas 
There are open sourced Types like cyberpandas for ip like objects and geopadnas for spatial like objects
Pandas Usage 🐼
Chunks 🍰
Splitting large data to smaller parts
Work with large datasets
End of explanation
%%timeit
df["meal_price_with_tip"].astype(object).mean()
%%timeit
df["meal_price_with_tip"].astype(float).mean()
Explanation: Mean/Sum/Mode/Min/etc optimization 🧮
Types matter
End of explanation
%%timeit
df[df.type_of_meal=="Breakfast"]
%%timeit
df.query("type_of_meal=='Breakfast'")
Explanation: 20x Performance Improvement⌛
DataFrame Serialization 🏋
Various file formats
Loading time
Saving time
Disk space
Query/Eval 🧬
Use numexpr if installed
Improve Execution Time 👍
The expected behaviour is up to 2 time faster
Improve Memory 👍
The main benefit of eval/query is the saved memory
NumPy allocate memory to every itermediate step.
and this Gives you the ability to compute this type of compound expression element by element, without the need to allocate full intermediate arrays.
Not all Operations are supported 👎
Example
End of explanation
def foo(N):
    accumulator = 0
    for i in range(N):
        accumulator = accumulator + i
    return accumulator
Explanation: % 20 Performance Improvement⌛
Use on big datasets 🧞
The pandas API indicate to only use eval on  more than 10,000 rows, as the traditional method is faster for smaller arrays.
Concat vs Append ➕
Every append creates a new dataframe object
Multiple appends are inefficient
Use concat
Sorting Optimization 📟
Vanilla python
Pandas 
Numpy
Pytorch/Tensorflow
GroupBy Optimizations 👩‍👩‍👧
Filter early
Custom functions are slow
Extract logic for custom functions when possible
Merge Optimization 🔍
Filter/Aggregate early
Semi join
Join on index
Compiled Code ?! Why 🤯
Python dynamic nature 
No compilation optimization
Pure Python can be slow
End of explanation
%%timeit
df.meal_price_with_tip.map(foo)
Explanation: Cython
the Cython project which converts Python code to compatiable C code
Numba
and the second one the Numba project, which converts Python code to fast LLVM bytecode
End of explanation
%load_ext Cython
Explanation: Cython and Numba for the rescue 👨‍🚒
Cython 🤯
Up to 50x speedup from pure python 👍
Learning Curve 👎
Separated Compilation Step 👎 👍
The compilation is both a con and a pro
on one hand, there is an additional work to integrate the cython code 
and on the other hand, there is no compilation overhead on runtime
End of explanation
%%cython
def cython_foo(long N):
    cdef long accumulator
    accumulator = 0
    cdef long i
    for i in range(N):
        accumulator += i
    return accumulator
%%timeit
df.meal_price_with_tip.map(cython_foo)
Explanation: Example
End of explanation
@jit(nopython=True)
def numba_foo(N):
    accumulator = 0
    for i in range(N):
        accumulator = accumulator + i
    return accumulator
%%timeit
df.meal_price_with_tip.map(numba_foo)
Explanation: 49x  Performance Improvement⌛
Numba 🤯
Up to 200x speedup from pure python  👍
Easy 👍
using numba is really easy its simply adding a decorator to a method
Highly Configurable  👍
Debugging 👎 👍
Debugging is both a con and a pro
on one hand, debugging the buissness logic is easy we can just remove the decorator and then we debug it as python code
on the other hand, if there is an issue in the compiled code itself its not quite easy to debug these issues.
Mostly Numeric 👎 
Example
End of explanation
@jit(nopython=True, parallel=True)
def numba_foo(N):
    accumulator = 0
    for i in range(N):
        accumulator = accumulator + i
    return accumulator
%%timeit
df.meal_price_with_tip.map(numba_foo)
Explanation: 43x  Performance Improvement⌛
End of explanation
def another_foo(data):
    return data * 2
def foo(data):
    return data + 10
Explanation: 1️⃣ Vectorized methods
2️⃣ Numba
3️⃣ Cython
General Python Optimizations 🐍
Since pandas is writen in python we can optimize our pandas code by using python optimization 
Caching 🏎
Avoid unnecessary work/computation.
Faster code 
Intermediate Variables👩‍👩‍👧‍👧
Intermediate calculations 
Memory foot print of both objects
Smarter variables allocation
End of explanation
%reload_ext memory_profiler
def load_data():
    return np.ones((2 ** 30), dtype=np.uint8)
%%memit
def proccess():
    data = load_data()
    return another_foo(foo(data))
proccess()
%%memit
def proccess():
    data = load_data()
    data = foo(data)
    data = another_foo(data)
    return data
proccess()
Explanation: Example
End of explanation
<END_TASK>