<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Demystifying pandas internals
Marc Garcia - @datapythonista
About me
<img style="float
Step1:  This is pandas, and it is super cool
What can go wrong?
And why should be care about how pandas works internally?
Step2:  ...you know you did something wrong
Why pandas? Why not pure Python?
Step3:  Why pandas? Why not pure Python?
Easy and powerful indexing
Management of missing values
High-level functionality
Step4:  Special numbers exist in floating point but not integer representation
Step5:  Example 1
Step6:  Computer architecture
Machine code
Address | Binary   | Decimal (uint8) |
--------|----------|-----------------|
0320    | 00001111 |             25  |
0321    | 00001010 |             41  |
0322    | 00000010 |              2  |
1) LOAD 0320 -&gt; R1   (R1=25, R2= ?)
2) LOAD 0321 -&gt; R2   (R1=25, R2=41)
3) ADD R1 R2 -&gt; R2   (R1=25, R2=66)
4) LOAD 0322 -&gt; R1   (R1= 2, R2=66)
5) ADD R1 R2 -&gt; R2   (R1= 2, R2=68)
Machine code
1) LOAD 0320 -&gt; R1   (R1=25, R2= ?)
2) LOAD 0321 -&gt; R2   (R1=25, R2=41)  &lt;- Preload to the CPU cache at 1)
3) ADD R1 R2 -&gt; R2   (R1=25, R2=66)
4) LOAD 0322 -&gt; R1   (R1= 2, R2=66)  &lt;- Preload to the CPU cache at 1)
5) ADD R1 R2 -&gt; R2   (R1= 2, R2=68)
Loading from the memory is slow
Step7:  Example 3
Step8:  numpy and pandas implementation are somehow similar to the machine code we've seen.
What about the pure Python implementation?
Python is implemented with great ideas in mind
Step9:  Address | PyObject                                                         |
--------|------------------------------------------------------------------|
0320    | ob_refcnt=1, ob_type=list, ob_size=3, ob_item=0510, allocated=16 |
 ...    | ...                                                              |
0510    |                                                           2478   |
0511    |                                                           5601   |
0512    |                                                           4882   |
 ...    | ...                                                              |
2478    | ob_refcnt=1, ob_type=int, ob_value=15                            |
 ...    | ...                                                              |
4882    | ob_refcnt=1, ob_type=int, ob_value=2                             |
 ...    | ...                                                              |
5601    | ob_refcnt=1, ob_type=int, ob_value=10                            |
1) LOAD 0320
2) LOAD 0510
3) LOAD 2478 (ob_value) -&gt; R1
4) LOAD 0511
5) LOAD 5601 (ob_value) -&gt; R2
6) ADD R1 R2 -&gt; R2
...
compare to
Step10:  Pandas internal structure
Step11:  Source
Step12:  Memory copy
Read/write to memory is slow
When a operation (e.g. a filter) is performed, a copy or a view can be returned
pandas returns a view in few cases, but is not deterministic
<ASSISTANT_TASK:>
Python Code:
import pandas
titanic = pandas.read_csv('data/titanic.csv')
titanic.head()
# basic statistics
titanic[['Age', 'Fare']].describe()
# percentage of missing values
titanic[['Age', 'Fare']].isna().mean()
# imputing missing values with the median
titanic['Age'].fillna(titanic['Age'].median(), inplace=True)
titanic[['Age', 'Fare']].isna().mean()
# plot distributions
titanic[['Age', 'Fare']].boxplot();
# dropping rows with 5% higher fares
titanic = titanic[titanic['Fare'] < titanic['Fare'].quantile(.95)]
titanic[['Age', 'Fare']].boxplot();
# plotting correlation
titanic.plot(kind='scatter', x='Age', y='Fare');
# compute correlation
titanic[['Age', 'Fare']].corr()
# get dummies for categories
dummies = pandas.get_dummies(titanic[['Sex', 'Embarked']])
dummies.head()
# prepare data for machine learning models
x = titanic[['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']].join(dummies)
y = titanic['Survived']
x.head()
import sklearn.ensemble
clf = sklearn.ensemble.RandomForestClassifier()
clf.fit(x, y)
pandas.Series(clf.feature_importances_, index=x.columns).sort_values(ascending=False)
titanic.plot(kind='scatter', x='Age', y='Fare', color=y.replace({0: 'red', 1: 'blue'}));
Explanation: Demystifying pandas internals
Marc Garcia - @datapythonista
About me
<img style="float: left; margin-right: 20px" src="img/mgarcia.png"/>
Marc Garcia
Pythonista since 2006
Master degree in AI
Pandas contributor
Organiser of the London Python Sprints group
Contractor data scientist
More info at my personal page and LinkedIn profile
Source code (jupyter notebook) of these slides:
- https://github.com/datapythonista/demystifying_pandas_internals
Overview
Quick pandas overview
What can go wrong?
Why pandas? Why not pure Python?
Data types
Computer architecture
pandas internal structure
Memory copy
Future of pandas
Quick pandas overview: Titanic
End of explanation
# some code resembling pandas to make the traceback look nice
import numpy
cols_from_csv = rows_from_csv = lambda fname: 10_000_000
def read_csv(fname, *args, **kwargs):
    cols = cols_from_csv(fname)
    rows = rows_from_csv(fname)
    create_block_manager_from_arrays(rows, cols)
def create_block_manager_from_arrays(rows, cols):
    float_blocks = _multi_blockify((rows, cols), numpy.float64)
def _multi_blockify(shape, dtype):
    _stack_arrays(shape, dtype)
def _stack_arrays(shape, dtype):
    stacked = numpy.empty(shape, dtype)
original_read_csv = pandas.read_csv
pandas.read_csv = read_csv
import pandas
df = pandas.read_csv('data/huge_file.csv')
pandas.read_csv = original_read_csv
original_read_csv = pandas.read_csv
class DataFrame:
    def __init__(self, *args, **kwargs):
        pass
    @staticmethod
    def sum():
        import IPython.display
        img = IPython.display.Image(filename='img/waiting.jpg')
        IPython.display.display(img)
pandas.read_csv = DataFrame
import pandas
df = pandas.read_csv('data/huge_file.csv')
total = df.sum()
pandas.read_csv = original_read_csv
import pandas
titanic = pandas.read_csv('data/titanic.csv')
titanic_male = titanic[titanic['Sex'] == 'male']
titanic_male['Age'].fillna(titanic_male['Age'].median(), inplace=True)
import warnings
import pandas
warnings.filterwarnings('ignore')
titanic = pandas.read_csv('data/titanic.csv')
titanic_male = titanic[titanic['Sex'] == 'male']
titanic_male['Age'].fillna(titanic_male['Age'].median(), inplace=True)
Explanation: This is pandas, and it is super cool
What can go wrong?
And why should be care about how pandas works internally?
End of explanation
# pandas.read_csv('data/titanic.csv')
import csv
data = []
with open('data/titanic.csv') as f:
    reader = csv.reader(f)
    columns = next(reader)
    for row in reader:
        data.append(row)
columns = ('Name', 'Sex', 'Age', 'Fare', 'Survived')
data = [('Montvila, Rev. Juozas',                    'male',     27.,    13., 0),
        ('Graham, Miss. Margaret Edith',             'female',   19.,    30., 1),
        ('Johnston, Miss. Catherine Helen "Carrie"', 'female',  None,  23.45, 0),
        ('Behr, Mr. Karl Howell',                    'male',     26.,    30., 1),
        ('Dooley, Mr. Patrick',                      'male',     32.,   7.75, 0)]
# df.iloc[3]
data[3]
# df['Fare']
fares = list(zip(*data))[columns.index('Fare')]
fares
# df['Fare'].mean()
mean_fare = sum(fares) / len(fares)
mean_fare
# df[df.Sex == 'male']
male = list(filter(lambda x: x[columns.index('Sex')] == 'male', data))
male
Explanation: ...you know you did something wrong
Why pandas? Why not pure Python?
End of explanation
0.1 + 0.2
import numpy
numpy.array([256], dtype=numpy.uint8)
Explanation: Why pandas? Why not pure Python?
Easy and powerful indexing
Management of missing values
High-level functionality:
Statistics
Time series
Groupping
...
Performance
Some computer engineering background
Data types
Computer architecture
Data types
unsigned integer 4 bits:
Binary | Decimal | | Binary | Decimal |
-------|---------|-|--------|---------|
0000   |      0  | | 1000   |      8  |
0001   |      1  | | 1001   |      9  |
0010   |      2  | | 1010   |     10  |
0011   |      3  | | 1011   |     11  |
0100   |      4  | | 1100   |     12  |
0101   |      5  | | 1101   |     13  |
0110   |      6  | | 1110   |     14  |
0111   |      7  | | 1111   |     15  |
Data types
signed integer 4 bits (two's complement):
Binary | Decimal | | Binary | Decimal |
-------|---------|-|--------|---------|
0000   |      0  | | 1000   |     -8  |
0001   |      1  | | 1001   |     -7  |
0010   |      2  | | 1010   |     -6  |
0011   |      3  | | 1011   |     -5  |
0100   |      4  | | 1100   |     -4  |
0101   |      5  | | 1101   |     -3  |
0110   |      6  | | 1110   |     -2  |
0111   |      7  | | 1111   |     -1  |
Data types
Decimal numbers:
 - Precision
 - Exponent
$value = precision \cdot 2^{exponent}$
$1328.543 = 1.328543 \cdot 10^{3}$
Data types
floating point 4 bits (2 bits exponent, 2 bits precision):
Binary  | Exponent | Precision | Decimal | | Binary  | Exponent | Precision | Decimal |
--------|----------|-----------|---------|-|---------|----------|-----------|---------|
00-00   |       0  |     1.00  |   1.00  | | 10-00   |      -2  |    1.00   |  0.010  |
00-01   |       0  |     1.50  |   1.50  | | 10-01   |      -2  |    1.50   |  0.015  |
00-10   |       0  |     1.25  |   1.25  | | 10-10   |      -2  |    1.25   |  0.013  |
00-11   |       0  |     1.75  |   1.75  | | 10-11   |      -2  |    1.75   |  0.018  |
01-00   |       1  |        0  |   +INF  | | 11-00   |      -1  |    1.00   |  0.100  |
01-01   |       1  |        1  |      0  | | 11-01   |      -1  |    1.50   |  0.150  |
01-10   |       1  |        2  |    NaN  | | 11-10   |      -1  |    1.25   |  0.125  |
01-11   |       1  |        3  |   -INF  | | 11-11   |      -1  |    1.75   |  0.175  |
Not all numbers can be represented
End of explanation
import pandas
s = pandas.Series([1, 2])
print(s)
s.loc[0] = float('NaN')
print(s)
Explanation: Special numbers exist in floating point but not integer representation: inf, -inf, NaN
End of explanation
import random
size = 10_000_000
list1 = [random.random() for i in range(size)]
list2 = [random.random() for i in range(size)]
import numpy
import pandas
series1 = pandas.Series(list1, dtype=numpy.uint8)
series2 = pandas.Series(list2, dtype=numpy.uint8)
print('Memory consumed: {:.2f} Mb'.format(series1.memory_usage(index=False) / 1024 / 1024))
%timeit (series1 > series2).mean()
import numpy
import pandas
series1 = pandas.Series(list1, dtype=numpy.float64)
series2 = pandas.Series(list2, dtype=numpy.float64)
print('Memory consumed: {:.2f} Mb'.format(series1.memory_usage(index=False) / 1024 / 1024))
%timeit (series1 > series2).mean()
Explanation: Example 1: Comparing list items, int vs float
End of explanation
import random
import functools
import operator
size = 10_000_000
list1 = [random.random() for i in range(size)]
%timeit functools.reduce(operator.add, list1)
import pandas
s1 = pandas.Series(list1)
%timeit s1.sum()
Explanation: Computer architecture
Machine code
Address | Binary   | Decimal (uint8) |
--------|----------|-----------------|
0320    | 00001111 |             25  |
0321    | 00001010 |             41  |
0322    | 00000010 |              2  |
1) LOAD 0320 -&gt; R1   (R1=25, R2= ?)
2) LOAD 0321 -&gt; R2   (R1=25, R2=41)
3) ADD R1 R2 -&gt; R2   (R1=25, R2=66)
4) LOAD 0322 -&gt; R1   (R1= 2, R2=66)
5) ADD R1 R2 -&gt; R2   (R1= 2, R2=68)
Machine code
1) LOAD 0320 -&gt; R1   (R1=25, R2= ?)
2) LOAD 0321 -&gt; R2   (R1=25, R2=41)  &lt;- Preload to the CPU cache at 1)
3) ADD R1 R2 -&gt; R2   (R1=25, R2=66)
4) LOAD 0322 -&gt; R1   (R1= 2, R2=66)  &lt;- Preload to the CPU cache at 1)
5) ADD R1 R2 -&gt; R2   (R1= 2, R2=68)
Loading from the memory is slow:
- CPU cache
Example 2: Summing list elements
End of explanation
%timeit sum(item1 > item2 for item1, item2 in zip(list1, list2)) / size
import pandas
series1 = pandas.Series(list1)
series2 = pandas.Series(list2)
%timeit (series1 > series2).mean()
Explanation: Example 3: Comparing list items, Python vs pandas
End of explanation
data = [1, 'foo', 3.141592, ['Alan', 'Dennis', 'Linus'],
        {'black': '#000000', 'white': '#ffffff'}]
data
Explanation: numpy and pandas implementation are somehow similar to the machine code we've seen.
What about the pure Python implementation?
Python is implemented with great ideas in mind:
- Readability
- Flexibility
- Hassle free
- No need to care about types
But not fast numerical computations
End of explanation
import pandas
pandas.Series(data)
import pandas
series1 = pandas.Series(list1, dtype='object')
series2 = pandas.Series(list2, dtype='object')
%timeit (series1 > series2).mean()
import numpy
import pandas
series1 = pandas.Series(list1, dtype=numpy.float64)
series2 = pandas.Series(list2, dtype=numpy.float64)
%timeit (series1 > series2).mean()
Explanation: Address | PyObject                                                         |
--------|------------------------------------------------------------------|
0320    | ob_refcnt=1, ob_type=list, ob_size=3, ob_item=0510, allocated=16 |
 ...    | ...                                                              |
0510    |                                                           2478   |
0511    |                                                           5601   |
0512    |                                                           4882   |
 ...    | ...                                                              |
2478    | ob_refcnt=1, ob_type=int, ob_value=15                            |
 ...    | ...                                                              |
4882    | ob_refcnt=1, ob_type=int, ob_value=2                             |
 ...    | ...                                                              |
5601    | ob_refcnt=1, ob_type=int, ob_value=10                            |
1) LOAD 0320
2) LOAD 0510
3) LOAD 2478 (ob_value) -&gt; R1
4) LOAD 0511
5) LOAD 5601 (ob_value) -&gt; R2
6) ADD R1 R2 -&gt; R2
...
compare to:
1) LOAD 0320 -&gt; R1   (R1=15, R2= ?)
2) LOAD 0321 -&gt; R2   (R1=15, R2=10)
3) ADD R1 R2 -&gt; R2   (R1=15, R2=25)
End of explanation
import pandas
df = pandas.DataFrame({'foo': [1, 3, 7],
                       'bar': [.55, 1.76, 3.33],
                       'foobar': [109, 60, 13]},
                      columns=['foo', 'bar', 'foobar'])
print(df.dtypes)
df
df._data
df._data.blocks
df._data.blocks[1].values
type(df._data.blocks[1].values)
df._data.blocks[1].values.data
df._data.blocks[1].values.nbytes
bytes_ = df._data.blocks[1].values.tobytes()
print(''.join('{:08b}'.format(byte) for byte in bytes_))
df._data.blocks[1].values.strides
Explanation: Pandas internal structure
End of explanation
import numpy
data = numpy.empty((1000, 1000), numpy.float64)
%timeit data.sum(axis=0)
%timeit data.sum(axis=1)
Explanation: Source: https://www.dataquest.io/blog/pandas-big-data/
End of explanation
import pandas
df = pandas.DataFrame({'foo': [1, 2, 3], 'bar': [5, 10, 15]})
df_view_or_copy = df[df.foo > 2]
df_view_or_copy['bar'] = 0  # are we modifying `df`? <- Chained indexing with unknown result: WARNING
df[df.foo > 2]['bar'] = 0  # same as before
df.loc[df.foo > 2, 'bar'] = 0  # we are modifying `df` (pandas manages __setitem__)
Explanation: Memory copy
Read/write to memory is slow
When a operation (e.g. a filter) is performed, a copy or a view can be returned
pandas returns a view in few cases, but is not deterministic
End of explanation
<END_TASK>