<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Tabular data
Step1:  Starting from reading this dataset, to answering questions about this data in a few lines of code
Step2:  How does the survival rate of the passengers differ between sexes?
Step3:  Or how does it differ between the different classes?
Step4:  Are young people more likely to survive?
Step5:  All the needed functionality for the above examples will be explained throughout this tutorial.
Data structures
Pandas provides two fundamental data objects, for 1D (Series) and 2D data (DataFrame).
Series
A Series is a basic holder for one-dimensional labeled data. It can be created much as a NumPy array is created
Step6:  Attributes of a Series
Step7:  You can access the underlying numpy array representation with the .values attribute
Step8:  We can access series values via the index, just like for NumPy arrays
Step9:  Unlike the NumPy array, though, this index can be something other than integers
Step10:  In this way, a Series object can be thought of as similar to an ordered dictionary mapping one typed value to another typed value.
In fact, it's possible to construct a series directly from a Python dictionary
Step11:  We can index the populations like a dict as expected
Step12:  but with the power of numpy arrays
Step13:  DataFrames
Step14:  Attributes of the DataFrame
A DataFrame has besides a index attribute, also a columns attribute
Step15:  To check the data types of the different columns
Step16:  An overview of that information can be given with the info() method
Step17:  Also a DataFrame has a values attribute, but attention
Step18:  If we don't like what the index looks like, we can reset it and set one of our columns
Step19:  To access a Series representing a column in the data, use typical indexing syntax
Step20:  Basic operations on Series/Dataframes
As you play around with DataFrames, you'll notice that many operations which work on NumPy arrays will also work on dataframes.
Step21:  Elementwise-operations (like numpy)
Just like with numpy arrays, many operations are element-wise
Step22:  Alignment! (unlike numpy)
Only, pay attention to alignment
Step23:  Reductions (like numpy)
The average population number
Step24:  The minimum area
Step25:  For dataframes, often only the numeric columns are included in the result
Step26:  <div class="alert alert-success">
    <b>EXERCISE</b>
Step27:  One useful method to use is the describe method, which computes summary statistics for each column
Step28:  The plot method can be used to quickly visualize the data in different ways
Step29:  However, for this dataset, it does not say that much
Step30:  You can play with the kind keyword
<ASSISTANT_TASK:>
Python Code:
df = pd.read_csv("data/titanic.csv")
df.head()
Explanation: Tabular data
End of explanation
df['Age'].hist()
Explanation: Starting from reading this dataset, to answering questions about this data in a few lines of code:
What is the age distribution of the passengers?
End of explanation
df.groupby('Sex')[['Survived']].aggregate(lambda x: x.sum() / len(x))
Explanation: How does the survival rate of the passengers differ between sexes?
End of explanation
df.groupby('Pclass')['Survived'].aggregate(lambda x: x.sum() / len(x)).plot(kind='bar')
Explanation: Or how does it differ between the different classes?
End of explanation
df['Survived'].sum() / df['Survived'].count()
df25 = df[df['Age'] <= 25]
df25['Survived'].sum() / len(df25['Survived'])
Explanation: Are young people more likely to survive?
End of explanation
s = pd.Series([0.1, 0.2, 0.3, 0.4])
s
Explanation: All the needed functionality for the above examples will be explained throughout this tutorial.
Data structures
Pandas provides two fundamental data objects, for 1D (Series) and 2D data (DataFrame).
Series
A Series is a basic holder for one-dimensional labeled data. It can be created much as a NumPy array is created:
End of explanation
s.index
Explanation: Attributes of a Series: index and values
The series has a built-in concept of an index, which by default is the numbers 0 through N - 1
End of explanation
s.values
Explanation: You can access the underlying numpy array representation with the .values attribute:
End of explanation
s[0]
Explanation: We can access series values via the index, just like for NumPy arrays:
End of explanation
s2 = pd.Series(np.arange(4), index=['a', 'b', 'c', 'd'])
s2
s2['c']
Explanation: Unlike the NumPy array, though, this index can be something other than integers:
End of explanation
pop_dict = {'Germany': 81.3, 
            'Belgium': 11.3, 
            'France': 64.3, 
            'United Kingdom': 64.9, 
            'Netherlands': 16.9}
population = pd.Series(pop_dict)
population
Explanation: In this way, a Series object can be thought of as similar to an ordered dictionary mapping one typed value to another typed value.
In fact, it's possible to construct a series directly from a Python dictionary:
End of explanation
population['France']
Explanation: We can index the populations like a dict as expected:
End of explanation
population * 1000
Explanation: but with the power of numpy arrays:
End of explanation
data = {'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
        'population': [11.3, 64.3, 81.3, 16.9, 64.9],
        'area': [30510, 671308, 357050, 41526, 244820],
        'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']}
countries = pd.DataFrame(data)
countries
Explanation: DataFrames: Multi-dimensional Data
A DataFrame is a tablular data structure (multi-dimensional object to hold labeled data) comprised of rows and columns, akin to a spreadsheet, database table, or R's data.frame object. You can think of it as multiple Series object which share the same index.
<img src="img/dataframe.png" width=110%>
One of the most common ways of creating a dataframe is from a dictionary of arrays or lists.
Note that in the IPython notebook, the dataframe will display in a rich HTML view:
End of explanation
countries.index
countries.columns
Explanation: Attributes of the DataFrame
A DataFrame has besides a index attribute, also a columns attribute:
End of explanation
countries.dtypes
Explanation: To check the data types of the different columns:
End of explanation
countries.info()
Explanation: An overview of that information can be given with the info() method:
End of explanation
countries.values
Explanation: Also a DataFrame has a values attribute, but attention: when you have heterogeneous data, all values will be upcasted:
End of explanation
countries = countries.set_index('country')
countries
Explanation: If we don't like what the index looks like, we can reset it and set one of our columns:
End of explanation
countries['area']
Explanation: To access a Series representing a column in the data, use typical indexing syntax:
End of explanation
# redefining the example objects
population = pd.Series({'Germany': 81.3, 'Belgium': 11.3, 'France': 64.3, 
                        'United Kingdom': 64.9, 'Netherlands': 16.9})
countries = pd.DataFrame({'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
        'population': [11.3, 64.3, 81.3, 16.9, 64.9],
        'area': [30510, 671308, 357050, 41526, 244820],
        'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']})
Explanation: Basic operations on Series/Dataframes
As you play around with DataFrames, you'll notice that many operations which work on NumPy arrays will also work on dataframes.
End of explanation
population / 100
countries['population'] / countries['area']
Explanation: Elementwise-operations (like numpy)
Just like with numpy arrays, many operations are element-wise:
End of explanation
s1 = population[['Belgium', 'France']]
s2 = population[['France', 'Germany']]
s1
s2
s1 + s2
Explanation: Alignment! (unlike numpy)
Only, pay attention to alignment: operations between series will align on the index:
End of explanation
population.mean()
Explanation: Reductions (like numpy)
The average population number:
End of explanation
countries['area'].min()
Explanation: The minimum area:
End of explanation
countries.median()
Explanation: For dataframes, often only the numeric columns are included in the result:
End of explanation
countries.sort_values('density', ascending=False)
Explanation: <div class="alert alert-success">
    <b>EXERCISE</b>: Calculate the population numbers relative to Belgium
</div>
<div class="alert alert-success">
    <b>EXERCISE</b>: Calculate the population density for each country and add this as a new column to the dataframe.
</div>
Some other useful methods
Sorting the rows of the DataFrame according to the values in a column:
End of explanation
countries.describe()
Explanation: One useful method to use is the describe method, which computes summary statistics for each column:
End of explanation
countries.plot()
Explanation: The plot method can be used to quickly visualize the data in different ways:
End of explanation
countries['population'].plot(kind='bar')
Explanation: However, for this dataset, it does not say that much:
End of explanation
pd.read
states.to
Explanation: You can play with the kind keyword: 'line', 'bar', 'hist', 'density', 'area', 'pie', 'scatter', 'hexbin'
Importing and exporting data
A wide range of input/output formats are natively supported by pandas:
CSV, text
SQL database
Excel
HDF5
json
html
pickle
...
End of explanation
<END_TASK>