<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <img style="float
Step1:  Let us take a sneak peek at the data
Step2:  What is the size of the dataset?
Step3:  Now we see that there are different models of hard disks, let us list them
<img style="float
Step4:  let us see how many models are there in total
Step5:  <img style="float
Step6:  Sort and plot
Step7:  Question 2. Find failures for a hard disk models
Step8:  Question 3. How do you compute failure rate for a model
Now let us express the failure / total number of hard disks as ratio. This will give us an understanding of models and their failure behavior
To get that data, instead of computing again, we can join the 2 data frames that were previously computed
and compute the ratio
Step9:  let us see in total hard disks for a model, how many failed and how many did not
Step10:  now let us compute the ratio of failure number/total_hard_disk of hard disk
Step11:  The higher the ratio value is , the model is prone to failure
Step12:  Now we know which models fail the most, let us introduce a new feature in our analysis, capacity. 
We are going feature by feature the reason being, the more features we add that add value to the outcome, we see how our understanding of the data starts to change.
 Let us look at the capacity
Step13:  Question 4. Given a model and capacity bytes, what does failure count look like
Step14:  Looking at this chart can you tell what is not being represented right?
We are having repeated entries for the same capacity and this really does not give us insights on the relation between capacity data and the models.
Step15:  we see that for some models and their respective capacitys we do not have a fail count, lets fill it with 0
Step16:  This heat map gives us a better understanding of model, capacity vs failure
Step17:  The above charts give us an explanation of which models failed the most, which models had the most number of hard disks running , the ratio of hard disk 
Step18:  Question 6. Find the average running time for failed hard disks and average running time for hard disks that have not failed
Step19:  <img style="float
Step20:  Now what can we do with this data? Is this useful? What can I generate from the above data that gives 
me a little more insight ?
We can generate what is the average time of failure and average success time for capacity
Step21:  Question 7. How about using hours (SMART_9) column now and co-relate it with failure
Step22:  Now we want to know upto when for a given hard disk and capacity , how long the hard disk ran
Step23:  Question 8. Given the data , identify the model and capacity of the hard disk to buy based on how long it runs
Step24:  Let us convert bytes to gigabytes and round it to the nearest number
Step25:  The above visualization is confusing as the bars reflect combination of failure and hours count
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams['figure.figsize']=15,10
df = pd.read_csv('data/data.csv')
Explanation: <img style="float:center" src="img/explore.jpg" width=300/>
Exploring the data
When we look at spreadsheets or large amounts of data, its hard for us to understand what is really happening. But when we visually interpret the data then everything starts making sense.
<img style="float::left" src="img/explore-reason.png" />
 Question 1. Find the total number of hard disks for a given model 
 Question 2. Find total failures for a hard disk models 
 Question 3. How do you compute failure rate for a model 
 Question 4. Given a model and capacity bytes, what does failure count look like
 Question 5. Let us count how many days each hard disk ran
 Question 6. Find the average running time for failed hard disks and average running time for hard disks that have not failed
 Question 7. How about using hours (SMART_9) column now and co-relate it with failure
 Question 8. Given the data , identify the model and capacity of the hard disk to buy based on how long it runs
Step by step approach
 First let us look at our data
End of explanation
df.head()
Explanation: Let us take a sneak peek at the data
End of explanation
df.shape
Explanation: What is the size of the dataset?
End of explanation
df_model = pd.DataFrame(df.model.unique(),columns=['model'])
df_model.head()
df_model.count()[0]
Explanation: Now we see that there are different models of hard disks, let us list them
<img style="float:center" src="img/distinct.gif" />
End of explanation
print "Total number of distinct models : "+ str(df_model.count()[0])
# Exerice 1: Find the distinct number of serial numbers
# Exercise 2: Find the distinct number of capacity bytes
Explanation: let us see how many models are there in total
End of explanation
df_model_serial = pd.DataFrame(df.groupby(['model']).serial.nunique())
df_model_serial.head()
df_model_serial = df_model_serial.reset_index()
df_model_serial.head()
df_model_serial.columns = ['model','total_HD']
df_model_serial.head(39)
df_model_serial.plot(kind="barh",x="model",y="total_HD")
Explanation: <img style="float:center" src="img/group-by.gif" />
Question 1. Find the total number of hard disks for a given model
Now let us see how many hard disks are there for each model and visualize it.
We see that serial number represents the hard disk and they are related to a model i.e multiple serial numbers belongs to one type of model
End of explanation
df_model_serial.sort_values(by='total_HD',inplace=True)
df_model_serial.plot(kind="barh",x="model",y="total_HD")
#Exercise 3: Find the count of different capacity bytes for a model and plot with and without sorting
Explanation: Sort and plot
End of explanation
df_fail = pd.DataFrame(df.groupby('model').failure.sum())
df_fail.head()
df_fail = df_fail.reset_index()
df_fail.head()
df_fail.plot(kind="barh",x="model",y="failure",figsize=(18,10))
# Exercise 4 : sort the above data frame and plot it
Explanation: Question 2. Find failures for a hard disk models
End of explanation
merged_df = df_model_serial.merge(df_fail,how='inner',on='model')
merged_df.head()
Explanation: Question 3. How do you compute failure rate for a model
Now let us express the failure / total number of hard disks as ratio. This will give us an understanding of models and their failure behavior
To get that data, instead of computing again, we can join the 2 data frames that were previously computed
and compute the ratio
End of explanation
merged_df['success'] = merged_df.total_HD - merged_df.failure
merged_df.head()
merged_df.plot(kind="bar",x="model",y=["failure","success"],subplots=True)
Explanation: let us see in total hard disks for a model, how many failed and how many did not
End of explanation
merged_df['ratio_failure'] = merged_df.failure / merged_df.total_HD 
merged_df.head(25)
merged_df.sort_values(by="ratio_failure",ascending=False,inplace=True)
merged_df.head()
merged_df.plot(kind="bar",x="model",y="ratio_failure")
Explanation: now let us compute the ratio of failure number/total_hard_disk of hard disk
End of explanation
#Exercise: Find ratio of success and plot it
#Exercise : Plot multiple bar charts comparing ratio of success and failure
Explanation: The higher the ratio value is , the model is prone to failure
End of explanation
df_capacity = pd.DataFrame(df.capacity.unique(),columns=['capacity'])
df_capacity.head()
df_capacity.shape
#Exercise : For a given capacity bytes, find the total number of failures and plot it
Explanation: Now we know which models fail the most, let us introduce a new feature in our analysis, capacity. 
We are going feature by feature the reason being, the more features we add that add value to the outcome, we see how our understanding of the data starts to change.
 Let us look at the capacity
End of explanation
df_fail_mod_cap = pd.DataFrame(df.groupby(['model','capacity']).failure.sum())
df_fail_mod_cap.head()
df_fail_mod_cap = df_fail_mod_cap.reset_index()
df_fail_mod_cap.head(25)
df_fail_mod_cap.plot(x="capacity",y="failure",kind="bar",figsize=(20,5))
Explanation: Question 4. Given a model and capacity bytes, what does failure count look like
End of explanation
df_fail_mod_cap.head()
df_fail_mod_cap_pivot = df_fail_mod_cap.pivot("model","capacity","failure")
df_fail_mod_cap_pivot.head()
Explanation: Looking at this chart can you tell what is not being represented right?
We are having repeated entries for the same capacity and this really does not give us insights on the relation between capacity data and the models.
End of explanation
df_fail_mod_cap.fillna(0,inplace=True)
df_fail_mod_cap.head()
sns.heatmap(df_fail_mod_cap_pivot)
Explanation: we see that for some models and their respective capacitys we do not have a fail count, lets fill it with 0
End of explanation
#Exercise : Find count of success for a model with different capacities and plot it
Explanation: This heat map gives us a better understanding of model, capacity vs failure
End of explanation
df_days = pd.DataFrame(df.groupby(['capacity','serial']).date.count())
df_days = df_days.reset_index()
df_days.head()
df_days.columns = ['capacity','serial','total_days']
df_days.head()
df_days.capacity.value_counts()
df_days.shape
df_days_pivot = df_days.pivot('capacity','serial','total_days')
df_days_pivot.head()
df_days_pivot.fillna(0,inplace=True)
df_days_pivot.head()
# Exercise : Visualize the above dataframe 
Explanation: The above charts give us an explanation of which models failed the most, which models had the most number of hard disks running , the ratio of hard disk : failure rate and hard disk and for a given capacity of a model what the failure count looks like
<img style="float:center" src="img/explore-clock.png" width=150/>
Hard disk data is time series data, so let us start using time
Question 5. Let us count how many days each hard disk ran
End of explanation
df_fail_days = pd.DataFrame(df[['capacity','serial','failure']].loc[df['failure'] == 1 ])
df_fail_days.head()
Explanation: Question 6. Find the average running time for failed hard disks and average running time for hard disks that have not failed
End of explanation
df_fail_count = df_days.merge(df_fail_days,how="left",on=['capacity','serial'])
df_fail_count.head()
df_fail_count.fillna(0,inplace=True)
df_fail_count.head()
df_fail_count.dtypes
g = sns.FacetGrid(df_fail_count, col="failure",hue='failure',size=5,aspect=1.5)
g.map_dataframe(plt.scatter,x='capacity',y='total_days')
Explanation: <img style="float:center" src="img/sql-joins.jpg"/>
now let us merge the previous data frame which had serial number and count of days
End of explanation
df_fail_count_avg = pd.DataFrame(df_fail_count.groupby(['capacity','failure']).total_days.mean())
df_fail_count_avg.head()
df_fail_count_avg = df_fail_count_avg.reset_index()
df_fail_count_avg.head()
df_fail_count_avg_pivot = df_fail_count_avg.pivot('capacity','failure','total_days')
df_fail_count_avg_pivot.head()
df_fail_count_avg_pivot.plot(kind="bar")
Explanation: Now what can we do with this data? Is this useful? What can I generate from the above data that gives 
me a little more insight ?
We can generate what is the average time of failure and average success time for capacity
End of explanation
df_hours = df[['serial','capacity','failure','smart_9']]
df_hours.head()
df_hours.shape
Explanation: Question 7. How about using hours (SMART_9) column now and co-relate it with failure
End of explanation
df_hours_max = pd.DataFrame(df_hours.groupby(['serial','capacity']).smart_9.max())
df_hours_max.head()
df_hours_max.shape
df_hours_max = df_hours_max.reset_index()
df_hours_max_merge = df_hours_max.merge(df_hours,on=['serial','capacity','smart_9'],how='inner')
df_hours_max_merge.head()
df_hours_max_merge_pivot = pd.pivot_table(df_hours_max_merge,index='capacity',columns='failure',values='smart_9'
                                          ,aggfunc='mean')
df_hours_max_merge_pivot.head()
df_hours_max_merge_pivot.plot(kind='bar')
Explanation: Now we want to know upto when for a given hard disk and capacity , how long the hard disk ran
End of explanation
df_model_capacity_hours = df[['model','capacity','failure','smart_9']]
df_model_capacity_hours.head()
Explanation: Question 8. Given the data , identify the model and capacity of the hard disk to buy based on how long it runs
End of explanation
df_model_capacity_hours.capacity = df_model_capacity_hours.capacity / 1024 ** 3
df_model_capacity_hours.head()
df_model_capacity_hours.capacity = df_model_capacity_hours.capacity.astype(np.int64)
df_model_capacity_hours.head()
df_model_capacity_hours_pivot = pd.pivot_table(data=df_model_capacity_hours,index='model',columns=['failure','capacity'],
                                               values='smart_9',aggfunc='mean')
df_model_capacity_hours_pivot.head()
df_model_capacity_hours_pivot.fillna(0,inplace=True)
df_model_capacity_hours_pivot.head()
df_model_capacity_hours_pivot.plot(kind="barh")
Explanation: Let us convert bytes to gigabytes and round it to the nearest number
End of explanation
sns.heatmap(df_model_capacity_hours_pivot)
Explanation: The above visualization is confusing as the bars reflect combination of failure and hours count
End of explanation
<END_TASK>