<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 4. Explore the Data
"I don't know, what I don't know"
We want to first visually explore the data to see if we can confirm some of our initial hypotheses as well as make new hypothesis about the problem we are trying to solve.
For this we will start by loading the data and understanding the data structure of the dataframe we have.
Lets read the data
Step1:  You will find the variable df used quite often to store a dataframe
Step2:  Understand Data Structure and Types
Step3:  Data Structure
So we have ten columns in our dataset. Let us understand what each one is.
Three are about the location of the Wholesale Market where Onion where sold.
- state
Step4:  So we are getting the quantitive columns are correctly being shown as integers and the categorical columns are showing as objects(strings) which is fine. 
However, the date columns is being read as an object and not a Time-Interval. Let us at least fix the date column and make it into a datetime object
Step5:  Question 1 - How big is the Bangalore onion market compared to other cities in India?
Let us try to do this examination for one of the year only. So we want to reduce our dataframe for only where the year = 2010. This process is called subset.
PRINCIPLE
Step6:  Principle
Step7:  PRINCIPLE
Step8:  Exercise
Step9:  Exercise
Step10:  PRINCIPLE
Step11:  To calculate the range of change, we will create a new price difference variable - which is the difference between the priceMin and priceMax
Step12:  PRINCIPLE
Step13:  Exercise
<ASSISTANT_TASK:>
Python Code:
# Import the library we need, which is Pandas
import pandas as pd
Explanation: 4. Explore the Data
"I don't know, what I don't know"
We want to first visually explore the data to see if we can confirm some of our initial hypotheses as well as make new hypothesis about the problem we are trying to solve.
For this we will start by loading the data and understanding the data structure of the dataframe we have.
Lets read the data
End of explanation
# Read the csv file of Monthwise Quantity and Price csv file we have.
df = pd.read_csv('MonthWiseMarketArrivals_clean.csv')
Explanation: You will find the variable df used quite often to store a dataframe
End of explanation
df.shape
df.head()
Explanation: Understand Data Structure and Types
End of explanation
# Get the typeof each column
df.dtypes
Explanation: Data Structure
So we have ten columns in our dataset. Let us understand what each one is.
Three are about the location of the Wholesale Market where Onion where sold.
- state: This is the 2/3 letter abbreviation for the state in India (PB is Punjab and so on)
- city: This is the city in India (ABOHAR, BANGALORE and so on)
- market: This is a string with the combination of the state and city
Three are related to the 
- month: Month in January, February and so on. 
- year: Year in YYYY representastion
- date: The combination of the two above.
Four are about quantity and price in these wholesale market. 
- quantity: The quanity of Onion arriving in the market in that month in quintals (100 kg)
- priceMin: The minimum price in the month in Rs./quintal
- priceMax: The maximum price in the month in Rs./quintal
- priceMod: The modal price in the month in Rs./quintal
We would expect the following the columns to be of the following type
- CATEGORICAL: state, city, market
- TIME INTERVAL: month, year, date
- QUANTITATIVE: quantity, priceMin, priceMax, priceModal
Let us see what pandas dataframe is reading these columns as.
End of explanation
# Changing the date column to a Time Interval columnn
df.date = pd.DatetimeIndex(df.date)
df.shape
# Now checking for type of each column
df.dtypes
# Let us see the dataframe again now
df.head()
# df.city.unique()
# Functional Approach
pd.unique(df.city)
Explanation: So we are getting the quantitive columns are correctly being shown as integers and the categorical columns are showing as objects(strings) which is fine. 
However, the date columns is being read as an object and not a Time-Interval. Let us at least fix the date column and make it into a datetime object
End of explanation
df2010 = df[df.year == 2010]
df2010.head()
# We can also subset on multiple criterias
df2010Bang = df[(df.year == 2010) & (df.city == 'BANGALORE')]
df2010Bang.head()
Explanation: Question 1 - How big is the Bangalore onion market compared to other cities in India?
Let us try to do this examination for one of the year only. So we want to reduce our dataframe for only where the year = 2010. This process is called subset.
PRINCIPLE: Subset a Dataframe
How do you subset a dataframe on a given criteria
newDataframe = df[ <subset condition> ]
End of explanation
# Group by using city
df2010City = df2010.groupby(['city']).sum()
df2010City.head()
type(df2010City)
# If we only want to apply the sum function on quantity, then we specify the quantity column
df2010City = df2010.groupby(['city']).quantity.sum()
# Let us see this dataframe
df2010City.head()
type(df2010City)
# To create a dataframe again, it is best to specify index as false
df2010City = df2010.groupby(['city'], as_index=False).quantity.sum()
df2010City.head()
sorted(df2010City.quantity)
# Sort the Dataframe by Quantity to see which one is on top
df2010City = df2010City.sort_values(by = "quantity", ascending = False)
df2010City.head()
%timeit sorted(df2010City.quantity)
%timeit df2010City.quantity.sort_values()
%timeit df2010City.sort_values(by = "quantity", ascending = False)
Explanation: Principle: Split Apply Combine
How do we get the sum of quantity for each city.
We need to SPLIT the data by each city, APPLY the sum to the quantity row and then COMBINE the data again
In pandas, we use the groupby function to do this.
End of explanation
# Load the visualisation libraries - Matplotlib
import matplotlib.pyplot as plt
# Let us see the output plots in the notebook itself
%matplotlib inline
# Set some parameters to get good visuals - style to ggplot and size to 15,10
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (15, 10)
# Plot the Data
df2010City.plot(kind ="barh", x = 'city', y = 'quantity')
Explanation: PRINCIPLE: Visual Exploration
Lets load the libraries required for plotting in python
End of explanation
df2015 = df[]
Explanation: Exercise: Find the State with Highest Quantity Sales in 2015?
End of explanation
df.head()
dfBang = df[df.city == 'BANGALORE']
dfBang.head()
dfBang.describe()
# Reduce the precision of numbers - so that it is easy to read
pd.set_option('precision', 0)
dfBang.describe()
Explanation: Exercise: Plot the State and Cities with Quantity Sales in 2015
Question 2 - Have the price variation in Onion prices in Bangalore really gone up over the years?
End of explanation
dfBang.head()
dfBang.index
# Set the index as date
dfBang = dfBang.sort_values(by = "date")
dfBang.head()
# Set the Index for the Dataframe
dfBang.index = pd.PeriodIndex(dfBang.date, freq='M')
dfBang.head()
dfBang.priceMod.plot()
dfBang.plot(kind = "line", y = ['priceMin', 'priceMod', 'priceMax'])
Explanation: PRINCIPLE: Setting Index
End of explanation
dfBang['priceDiff'] = dfBang['priceMax'] - dfBang['priceMin']
dfBang.head()
dfBang.plot(kind = 'line', y = 'priceDiff')
Explanation: To calculate the range of change, we will create a new price difference variable - which is the difference between the priceMin and priceMax
End of explanation
# Create new variable for Integer Month
dfBang['monthVal'] = pd.DatetimeIndex(dfBang['date']).month
dfBang.head()
dfBangPivot = pd.pivot_table(dfBang, values = "priceDiff", 
                             columns = "year", index = "monthVal")
dfBangPivot
dfBangPivot.plot()
dfBangPivot.plot(subplots = True, figsize=(15, 15), layout=(3, 5), sharey=True)
Explanation: PRINCIPLE: Pivot Table
Pivot table is a way to summarize data frame data into index (rows), columns and value
End of explanation
dfBangPivot.plot(subplots = True, figsize=(15, 15), layout=(3, 5), 
                 sharey=False)
Explanation: Exercise: Find the price variation for LASALGAON city?
End of explanation
<END_TASK>