<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Download
Goal of Week 2
This week we will
Step1:  Let's read in some data. Our file is an .xls file and it has 2 sheets. I don't know how to do that, so I will google "pandas read in excel file multiple sheets" and see what happens.
First useful result came up was
Step2:  Let's take a look at the "tables" we read in. By the way, in Python pandas terms, the data structure of the data we read in is called a DataFrame. This is the reason why when you search for answers online, a lot of people will refer to their variable names as df.
Step3:  That's beautiful. Now pandas has a nice function called head() that summarizes our data. Right now our data only has 2 columns and maybe 10+ rows. But head() is going to be super useful to have a giant table of 1000 rows x 1000 columns. This is how you use it.
Step4:  If you search up the documentation, you will see we can pass in an argument of how many rows we want to show. Let's see if we can show the first 7 rows.
Step5:  That works! Let's now doing some fun stuff with pandas. I want to rename the two columns in liquid_table_temp to "Primary" and "Ultrasonic". Let's take a look at the current column names first. As always, I googled "list of column names" and found this Stackoverflow answer.
Step6:  Cool. Let's rename it now. Again, I googled "pandas rename column" and found http
Step7:  Let's check it now
Step8:  Let's say I, for some reasons, want to add the primary + ultrasonic together and create a new column called mySum. Again, googled for "pandas add column together" and found http
Step9:  Let's say now I want to create a new column that is the cumulative sum of Ultrasonic.
Step10:  Now let's say I want to divide this CumulativeSum by the maximum of that Cumulative Sum column. I'm going to call the new column something arbitrary, "myDivision".
Again, I don't know how to find the max of the CumulativeSum column so I googled "pandas max value in column". Found this http
Step11:  Now let's say I want to sort the table based on the Primary column from smallest to largest.
Step12:  Apparently the sort() we are trying to use is deprecreted, I don't know what that quite mean but I'm gonna do what it says and use sort_values() instead
Step13:  Now let's do some filtering. I want my table to only shows Primary values that are greater than 40. Let's first take a look at the original table before we move on.
Step14:  http
Step15:  That stackoverflow answer brought up a really nice option. That is, how can I select data where only certain values is what I want. For example, I want the table to only show rows where my Primary is either 39.61 or 60.64.
Step16:  Not working so I searched "pandas isin()" and found http
Step17:  Great! That sounds like most of the things I want to do. Now I want to try somethinga litle bit different and actually plot them out. I'm going back to my temp_table and plot the K-type and T-Type. Let's take a look at the temp_table again before we start.
Step18:  Now I frankly don't know where to start so I googled "how to plot with python". First result came up was http
Step19:  Now what if I don't want the figure to come up every time? Is there a way for Jupyter notebook to display the images within the notebook I'm working on?
Yes there is.
Step20:  Now I'm going to run that again and see what happens.
Step21:  Perfect! Let's try now to plot our data.
Step22:  But now I also want the other column in the same plot! No problem. Just add another line.
Step23:  Awesome! Now I want to add in the title for the x-axis, y-axis, figure, label, and legend.
Step24:  Sounds great! Now let's go a little bit further and decorate our plots. With some googling, "matplotlib change line styles" gave me http
Step25:  Awesome. Let's say we have to plot the same thing over and over again for 10 different columns. That's going to be very annoying to do in Excel. So let's see if we can write up some function and "automate" this for us. I'm going to copy&paste all the code above and put it in a function.
Step26:  Now I'm going plot several columns in the liquid_flow_table (because there are many columns). Again, let's take a look at the table again.
Step27:  So 5 different plots, all in 5 lines! Imagine doing this in Excel! And in fact, programming is so awesome you can in fact not even have to copy&paste and do all that work.
Step28:  So if you're not still sold over this, I don't know what will 
<ASSISTANT_TASK:>
Python Code:
import pandas as pd
Explanation: Download
Goal of Week 2
This week we will:
Take a look at pandas, an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools
Load in Excel (.xls or .csv) files to use in Python
Managing headers
Read-in only certain columns
Do some data manipulation:
List the names of the columns
Renaming columns
Create a new column
Doing math operations between two columns
Filtering out data or a subset of data
Do some plotting using Matplotlib, a very nice 2D plotting library in Python
Plot a histogram
Plot scatter plots
Plot normal plots
Modify the x-axis's title, y-axis's title, figure's title
If we have time, we can go through a quick example to show how powerful pandas is when it renders tens of thousands of rows super quickly.
Note
Remember to run each invidual cell you can hit Ctrl+Enter
If you want to add a cell above or below, you can hit Esc + A or Esc + B. Or you can go to the toolbar at the top -> Insert -> Insert Cell Above/Insert Cell Below
If you want to run all the cells altogether, go to the toolbar -> Cell -> Run All.
In the case something catastrophic happens and everything explodes, keep calmn, go to the the toolbar -> Kernal -> Restart
Let's get started
Before we start, let's first import the pandas package. The syntax is typically:
python
import nameOfPackage
However, whenever you import a package and call a function from that package, you will have to write explicitly the name of the package. So I'm gonna rename the pandas package to something shorter, like pd.
python
import pandas as pd
The "as" syntax will allow me to refer to the pandas package as pd from now on.
End of explanation
xls = pd.ExcelFile(r'C:\Users\jenng\Documents\texaspse-blog\media\f16-scientific-python\week2\myExcelData.xls')
temp_table = xls.parse('Temperature')
liquid_flow_table = xls.parse('Liquid Flow')
Explanation: Let's read in some data. Our file is an .xls file and it has 2 sheets. I don't know how to do that, so I will google "pandas read in excel file multiple sheets" and see what happens.
First useful result came up was: http://stackoverflow.com/questions/26521266/using-pandas-to-pd-read-excel-for-multiple-worksheets-of-the-same-workbook so I'm gonna do what the upvoted did.
End of explanation
temp_table
Explanation: Let's take a look at the "tables" we read in. By the way, in Python pandas terms, the data structure of the data we read in is called a DataFrame. This is the reason why when you search for answers online, a lot of people will refer to their variable names as df.
End of explanation
temp_table.head()
Explanation: That's beautiful. Now pandas has a nice function called head() that summarizes our data. Right now our data only has 2 columns and maybe 10+ rows. But head() is going to be super useful to have a giant table of 1000 rows x 1000 columns. This is how you use it.
End of explanation
temp_table.head(7)
Explanation: If you search up the documentation, you will see we can pass in an argument of how many rows we want to show. Let's see if we can show the first 7 rows.
End of explanation
list(liquid_flow_table.columns.values)
Explanation: That works! Let's now doing some fun stuff with pandas. I want to rename the two columns in liquid_table_temp to "Primary" and "Ultrasonic". Let's take a look at the current column names first. As always, I googled "list of column names" and found this Stackoverflow answer.
End of explanation
liquid_flow_table = liquid_flow_table.rename(columns={'Primary Std, Avg mL/s': 'Primary', 
                        'Ultrasonic, mL/S': 'Ultrasonic',})
Explanation: Cool. Let's rename it now. Again, I googled "pandas rename column" and found http://stackoverflow.com/questions/11346283/renaming-columns-in-pandas
End of explanation
liquid_flow_table.head() # Great! It worked like we wanted.
Explanation: Let's check it now:
End of explanation
liquid_flow_table['mySum'] = liquid_flow_table.Primary + liquid_flow_table.Ultrasonic
liquid_flow_table.head() #check my table to see if it did add
liquid_flow_table['myMultiplication'] = liquid_flow_table['Primary'] * liquid_flow_table['Ultrasonic']
liquid_flow_table.head() #check my table to see if it did multiply
liquid_flow_table['myPower'] = liquid_flow_table.Primary ** liquid_flow_table.Ultrasonic
liquid_flow_table.head() #check my table to see if it did multiply
Explanation: Let's say I, for some reasons, want to add the primary + ultrasonic together and create a new column called mySum. Again, googled for "pandas add column together" and found http://stackoverflow.com/questions/22342285/summing-two-columns-in-a-pandas-dataframe
End of explanation
liquid_flow_table['CumulativeSum'] = liquid_flow_table.Ultrasonic.cumsum()
liquid_flow_table
Explanation: Let's say now I want to create a new column that is the cumulative sum of Ultrasonic.
End of explanation
liquid_flow_table['CumulativeSum'].max() #looks right
liquid_flow_table['myDivision'] = liquid_flow_table['CumulativeSum']/liquid_flow_table['CumulativeSum'].max()
liquid_flow_table.head()
Explanation: Now let's say I want to divide this CumulativeSum by the maximum of that Cumulative Sum column. I'm going to call the new column something arbitrary, "myDivision".
Again, I don't know how to find the max of the CumulativeSum column so I googled "pandas max value in column". Found this http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.max.html
End of explanation
liquid_flow_table.sort('Primary', ascending=True, inplace=True)
Explanation: Now let's say I want to sort the table based on the Primary column from smallest to largest.
End of explanation
liquid_flow_table.sort_values(by='Primary', ascending=True, inplace=True)
liquid_flow_table.head()
Explanation: Apparently the sort() we are trying to use is deprecreted, I don't know what that quite mean but I'm gonna do what it says and use sort_values() instead
End of explanation
liquid_flow_table
Explanation: Now let's do some filtering. I want my table to only shows Primary values that are greater than 40. Let's first take a look at the original table before we move on.
End of explanation
liquid_flow_table.loc[liquid_flow_table['Ultrasonic'] > 40].head()
Explanation: http://stackoverflow.com/questions/17071871/select-rows-from-a-dataframe-based-on-values-in-a-column-in-pandas
End of explanation
liquid_flow_table.loc[liquid_flow_table['Primary'].isin(39.62,60.64)].head()
Explanation: That stackoverflow answer brought up a really nice option. That is, how can I select data where only certain values is what I want. For example, I want the table to only show rows where my Primary is either 39.61 or 60.64.
End of explanation
liquid_flow_table.loc[liquid_flow_table['Primary'].isin([39.62,60.64])].head()
Explanation: Not working so I searched "pandas isin()" and found http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.isin.html
Seems like they put their values in [] so I'm gonna do that and see if it fixes the problem.
End of explanation
temp_table
Explanation: Great! That sounds like most of the things I want to do. Now I want to try somethinga litle bit different and actually plot them out. I'm going back to my temp_table and plot the K-type and T-Type. Let's take a look at the temp_table again before we start.
End of explanation
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
plt.ylabel('some numbers')
plt.show()
Explanation: Now I frankly don't know where to start so I googled "how to plot with python". First result came up was http://matplotlib.org/users/pyplot_tutorial.html. I'm just gonna plug in what the first box says.
End of explanation
%matplotlib inline
Explanation: Now what if I don't want the figure to come up every time? Is there a way for Jupyter notebook to display the images within the notebook I'm working on?
Yes there is.
End of explanation
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
plt.ylabel('some numbers')
plt.show()
Explanation: Now I'm going to run that again and see what happens.
End of explanation
# no need for the import statement again because it's already imported
plt.plot(temp_table['K-type TC, °C'])
plt.ylabel('some numbers')
plt.show()
Explanation: Perfect! Let's try now to plot our data.
End of explanation
plt.plot(temp_table['K-type TC, °C'])
plt.plot(temp_table['T-type TC, °C'])
plt.ylabel('some numbers')
plt.show()
Explanation: But now I also want the other column in the same plot! No problem. Just add another line.
End of explanation
plt.plot(temp_table['K-type TC, °C'])
plt.plot(temp_table['T-type TC, °C'])
plt.ylabel('Temperature, °C')
plt.xlabel('Index')
plt.title('Temperature vs. Index')
plt.legend()
plt.show()
Explanation: Awesome! Now I want to add in the title for the x-axis, y-axis, figure, label, and legend.
End of explanation
plt.plot(temp_table['K-type TC, °C'],'-.',color='r',linewidth=3)
plt.plot(temp_table['T-type TC, °C'],'o-',color='c')
plt.ylabel('Temperature, °C')
plt.xlabel('Index')
plt.title('Temperature vs. Index')
plt.legend()
plt.show()
Explanation: Sounds great! Now let's go a little bit further and decorate our plots. With some googling, "matplotlib change line styles" gave me http://matplotlib.org/examples/lines_bars_and_markers/line_styles_reference.html
End of explanation
def plotMyData(x,y):
    plt.plot(x, y,'-.',color='r',linewidth=3)
    plt.xlabel(x.name)
    plt.ylabel(y.name)
    plt.title(y.name + " vs. " + x.name)
    plt.legend()
    plt.show()
Explanation: Awesome. Let's say we have to plot the same thing over and over again for 10 different columns. That's going to be very annoying to do in Excel. So let's see if we can write up some function and "automate" this for us. I'm going to copy&paste all the code above and put it in a function.
End of explanation
liquid_flow_table.head()
plotMyData(liquid_flow_table.Primary,liquid_flow_table.mySum)
plotMyData(liquid_flow_table.Primary,liquid_flow_table.myMultiplication)
plotMyData(liquid_flow_table.Primary,liquid_flow_table.myPower)
plotMyData(liquid_flow_table.Primary,liquid_flow_table.CumulativeSum)b
plotMyData(liquid_flow_table.Primary,liquid_flow_table.myDivision)
Explanation: Now I'm going plot several columns in the liquid_flow_table (because there are many columns). Again, let's take a look at the table again.
End of explanation
liquid_table_columns_list = list(liquid_flow_table.columns.values)
liquid_table_columns_list
for column in liquid_table_columns_list:
    plotMyData(liquid_flow_table.Primary, liquid_flow_table[column])b
Explanation: So 5 different plots, all in 5 lines! Imagine doing this in Excel! And in fact, programming is so awesome you can in fact not even have to copy&paste and do all that work.
End of explanation
%%timeit
xls = pd.read_csv(r'C:\Users\jenng\Documents\texaspse-blog\media\f16-scientific-python\week2\GASISData.csv')
Explanation: So if you're not still sold over this, I don't know what will :)
If we have time, let's open our GASISData. If you open it in excel, you can see that it's a huge table. About 19220 rows and maybe 60 columns. Let's read the file in and see how pandas handle it.
I'm going to use something that's specific to Jupyter Notebook to time how long it reads in the data.
End of explanation
<END_TASK>