<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Short demo of using ipython_memory_usage to diagnose numpy and Pandas RAM usage
Author Ian uses this tool in his Higher Performance Python training (https
Step1:  Importing packages uses some RAM
Step2:  Making a large array uses a predictable amount of RAM
Step3:  Making a big random array takes RAM + time
Step4:  Intermediate calculations can cost additional temporary RAM
NOTE this section may work different if you're on Windows (if so - please report back to Ian by raising a bug and noting the difference.
On some platforms, e.g. Linux as used here, temporary intermediates can be reused in-place reducing the overall memory allocation
Step5:  Pandas DataFrames can be costly on RAM
Example with deleting columns
Props to Jamie Brunning for this example
Step6:  Diagnostics
%xdel my_df will delete all references of my_df from the namespace including those in the Out[] history buffer, this does more cleaning than just using del my_df.
%reset will reset all variables and imported modules, it is like starting a new kernel.
<ASSISTANT_TASK:>
Python Code:
import ipython_memory_usage
help(ipython_memory_usage) # or ipython_memory_usage?
%ipython_memory_usage_start
Explanation: Short demo of using ipython_memory_usage to diagnose numpy and Pandas RAM usage
Author Ian uses this tool in his Higher Performance Python training (https://ianozsvald.com/training/) and it is mentioned in his High Performance Python (2nd ed, O'Reilly) technical book.
We can use it to understand how much RAM we're currently using and which of several alternate ways to solve a problem in complex tools might be the most RAM efficient solutions.
total RAM usage is the current RAM usage at the end of that cell's execution
used shows the difference between the last total RAM usage and this one
peaked shows any during-execution peak above the resulting total RAM usage (i.e. hidden RAM usage that might catch you out)
End of explanation
import numpy as np # note that importing a package will increase total RAM usage a little
import pandas as pd # note that importing Pandas uses more RAM than importing numpy
import string
Explanation: Importing packages uses some RAM
End of explanation
# if we make a big array - 100M items * 8 byte floats, this cell
# uses circa 800MB (often 760 MiB - note mibi-bytes as used in the underlying memory_profiler tool)
# The total RAM usage grows by roughly this amount
arr = np.ones(100_000_000) 
# deleting arr reduces RAM usage by roughly the expected amount and
# total RAM usage should drop back down
del arr
# if we make it again, RAM usage goes up again
arr = np.ones(100_000_000) 
del arr
Explanation: Making a large array uses a predictable amount of RAM
End of explanation
# creating random items takes some time, after "used ... RAM" note "3s" or so for several seconds
arr = np.random.normal(size=100_000_000)
print(arr[:5], arr.dtype)
Explanation: Making a big random array takes RAM + time
End of explanation
pass
# arr*2 and arr*3 both have to be stored somewhere before the division can occur
# so two more circa 762MiB arrays are made temporarily, this is reported
# as "peaked 762MiB above current"
# before they can be discard. arr_result references the final result
# so overall we add 762MiB to the process
# we only add 762MiB, not 762MiB*2, as on Linux we can intelligently reuse
# one of the temporaries (else we'd peak at 762*2 MiB)
# we report "used 762...MiB" as the final arr_result adds this to the process
# so overall we're now _at_ 1.6GB but we actually peaked at 1.6+0.7 == 2.3GB 
# whilst this cell executed
# if your code crashes with an out of memory exception, it could be caused
# by a situation like this
arr_result = (arr * 2) / (arr * 3)
del arr
del arr_result
Explanation: Intermediate calculations can cost additional temporary RAM
NOTE this section may work different if you're on Windows (if so - please report back to Ian by raising a bug and noting the difference.
On some platforms, e.g. Linux as used here, temporary intermediates can be reused in-place reducing the overall memory allocation: https://docs.scipy.org/doc/numpy-1.13.0/release.html#highlights
End of explanation
pass
arr_several_cols = np.random.normal(size=(100_000_000, 4))
arr_several_cols.shape
f"Cost per column {int(arr_several_cols.data.nbytes / arr_several_cols.shape[1]):,} bytes"
# The DataFrame in this case is a thin wrapper over the numpy array
# and costs little extra RAM
df = pd.DataFrame(arr_several_cols, columns=list(string.ascii_lowercase)[:arr_several_cols.shape[1]])
df.info()
# use Jupyter's xdel to remove all references of our expensive array, just in case
# (but not in this case) it is also referred to in an Out[] history item
%xdel arr_several_cols
df.info()
# using del is surprisingly expensive
# total RAM usage goes up by circa 1.5GB-2GB (>2x the cost of 1 column) 
# DOES ANYONE KNOW WHAT'S HAPPENING BEHIND THE SCENES HERE?
# THE NEXT 2 CELLS SHOW IT ISN'T BEING QUICKLY GARBAGE COLLECTED
# note also that using del seems to take more seconds than using df.drop (a few cells below)
# possibly internally there's now (somehow) a 4-column original array _and_ a
# 3 column resulting array (in the BlockManager?) costing 7-columns (i.e. circa 800MB*7 == circa 5.6GB)
del df['a']
# we get no benefit by forcing a collection
import gc
gc.collect()
df.info()
pass
# using drop with inplace=False (the default) returns a copied DataFrame, if you don't use
# this then maybe you end up with multiple DataFrames consuming RAM in a confusing fashion
# e.g. you might have done `df2 = df.drop...` and then you've got the unmodified original
# plus the modified df2 in the local namespace
# We see total RAM usage drop by circa 800MB, the cost of 1 column, plus a lot more...
# which is a mystery to me!
# maybe the usage of drop forces a flush on any internal caching in pandas?
df = df.drop(columns=['b'])
df.info()
# dropping in-place is probably more sensible, we recover another circa 800MB
df.drop(columns=['c'], inplace=True)
df.info()
pass
# now we get back to where we were before we made the DataFrame and the array
df.drop(columns=['d'], inplace=True)
Explanation: Pandas DataFrames can be costly on RAM
Example with deleting columns
Props to Jamie Brunning for this example
End of explanation
# %whos shows what's in the local namespace
%whos
# we can use %xdel to safely remove all references including those that might be (but not in this case)
# in the Out[] history buffer
%xdel df
%ipython_memory_usage_stop
Explanation: Diagnostics
%xdel my_df will delete all references of my_df from the namespace including those in the Out[] history buffer, this does more cleaning than just using del my_df.
%reset will reset all variables and imported modules, it is like starting a new kernel.
End of explanation
<END_TASK>