<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 02 - Introduction to Python for Data Analysis
by Alejandro Correa Bahnsen and Jesus Solano
version 1.4, January 2019
Part of the class Practical Machine Learning
This notebook is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. Special thanks goes to Rick Muller, Sandia National Laboratories
Why Python?
Python is the programming language of choice for many scientists to a large degree because it offers a great deal of power to analyze and model scientific data with relatively little overhead in terms of learning, installation or development time. It is a language you can pick up in a weekend, and use for the rest of one's life.
The Python Tutorial is a great place to start getting a feel for the language. To complement this material, I taught a Python Short Course years ago to a group of computational chemists during a time that I was worried the field was moving too much in the direction of using canned software rather than developing one's own methods. I wanted to focus on what working scientists needed to be more productive
Step1:  I. Python Overview
This is a quick introduction to Python. There are lots of other places to learn the language more thoroughly. I have collected a list of useful links, including ones to other learning resources, at the end of this notebook. If you want a little more depth, Python Tutorial is a great place to start, as is Zed Shaw's Learn Python the Hard Way.
The lessons that follow make use of the IPython notebooks. There's a good introduction to notebooks in the IPython notebook documentation that even has a nice video on how to use the notebooks. You should probably also flip through the IPython tutorial in your copious free time.
Briefly, notebooks have code cells (that are generally followed by result cells) and text cells. The text cells are the stuff that you're reading now. The code cells start with "In []
Step2:  (If you're typing this into an IPython notebook, or otherwise using notebook file, you hit shift-Enter to evaluate a cell.)
In the last few lines, we have sped by a lot of things that we should stop for a moment and explore a little more fully. We've seen, however briefly, two different data types
Step3:  or you can simply import the math library itself
Step4:  You can define variables using the equals (=) sign
Step5:  You can name a variable almost anything you want. It needs to start with an alphabetical character or "_", can contain alphanumeric characters plus underscores ("_"). Certain words, however, are reserved for the language
Step6:  The Python Tutorial has more on using Python as an interactive shell. The IPython tutorial makes a nice complement to this, since IPython has a much more sophisticated iteractive shell.
Strings
Strings are lists of printable characters, and can be defined using either single quotes
Step7:  or double quotes
Step8:  Just like the other two data objects we're familiar with (ints and floats), you can assign a string to a variable
Step9:  The print statement is often used for printing character strings
Step10:  But it can also print data types other than strings
Step11:  In the above snipped, the number 600 (stored in the variable "area") is converted into a string before being printed out.
You can use the + operator to concatenate strings together
Step12:  If you have a lot of words to concatenate together, there are other, more efficient ways to do this. But this is fine for linking a few strings together.
Lists
Very often in a programming language, one wants to keep a group of similar items together. Python does this using a data type called lists.
Step13:  You can access members of the list using the index of that item
Step14:  Python lists, like C, but unlike Fortran, use 0 as the index of the first element of a list. Thus, in this example, the 0 element is "Sunday", 1 is "Monday", and so on. If you need to access the nth element from the end of the list, you can use a negative index. For example, the -1 element of a list is the last element
Step15:  You can add additional items to the list using the .append() command
Step16:  The range() command is a convenient way to make sequential lists of numbers
Step17:  Note that range(n) starts at 0 and gives the sequential list of integers less than n. If you want to start at a different number, use range(start,stop)
Step18:  The lists created above with range have a step of 1 between elements. You can also give a fixed step size via a third command
Step19:  Lists do not have to hold the same data type. For example,
Step20:  However, it's good (but not essential) to use lists for similar objects that are somehow logically connected. If you want to group different data types together into a composite data object, it's best to use tuples, which we will learn about below.
You can find out how long a list is using the len() command
Step21:  Iteration, Indentation, and Blocks
One of the most useful things you can do with lists is to iterate through them, i.e. to go through each element one at a time. To do this in Python, we use the for statement
Step22:  This code snippet goes through each element of the list called days_of_the_week and assigns it to the variable day. It then executes everything in the indented block (in this case only one line of code, the print statement) using those variable assignments. When the program has gone through every element of the list, it exists the block.
(Almost) every programming language defines blocks of code in some way. In Fortran, one uses END statements (ENDDO, ENDIF, etc.) to define code blocks. In C, C++, and Perl, one uses curly braces {} to define these blocks.
Python uses a colon ("
Step23:  The range() command is particularly useful with the for statement to execute loops of a specified length
Step24:  Slicing
Lists and strings have something in common that you might not suspect
Step25:  This is only occasionally useful. Slightly more useful is the slicing operation, which you can also use on any sequence. We already know that we can use indexing to get the first element of a list
Step26:  If we want the list containing the first two elements of a list, we can do this via
Step27:  or simply
Step28:  If we want the last items of the list, we can do this with negative slicing
Step29:  which is somewhat logically consistent with negative indices accessing the last elements of the list.
You can do
Step30:  Since strings are sequences, you can also do this to them
Step31:  If we really want to get fancy, we can pass a third element into the slice, which specifies a step length (just like a third argument to the range() function specifies the step)
Step32:  Note that in this example I was even able to omit the second argument, so that the slice started at 2, went to the end of the list, and took every second element, to generate the list of even numbers less that 40.
Booleans and Truth Testing
We have now learned a few data types. We have integers and floating point numbers, strings, and lists to contain them. We have also learned about lists, a container that can hold any data type. We have learned to print things out, and to iterate over items in lists. We will now learn about boolean variables that can be either True or False.
We invariably need some concept of conditions in programming to control branching behavior, to allow a program to react differently to different situations. If it's Monday, I'll go to work, but if it's Sunday, I'll sleep in. To do this in Python, we use a combination of boolean variables, which evaluate to either True or False, and if statements, that control branching based on boolean values.
For example
Step33:  (Quick quiz
Step34:  If we evaluate it by itself, as we just did, we see that it returns a boolean value, False. The "==" operator performs equality testing. If the two items are equal, it returns True, otherwise it returns False. In this case, it is comparing two variables, the string "Sunday", and whatever is stored in the variable "day", which, in this case, is the other string "Saturday". Since the two strings are not equal to each other, the truth test has the false value.
The if statement that contains the truth test is followed by a code block (a colon followed by an indented block of code). If the boolean is true, it executes the code in that block. Since it is false in the above example, we don't see that code executed.
The first block of code is followed by an else statement, which is executed if nothing else in the above if statement is true. Since the value was false, this code is executed, which is why we see "Go to work".
You can compare any data types in Python
Step35:  We see a few other boolean operators here, all of which which should be self-explanatory. Less than, equality, non-equality, and so on.
Particularly interesting is the 1 == 1.0 test, which is true, since even though the two objects are different data types (integer and floating point number), they have the same value. There is another boolean operator is, that tests whether two objects are the same object
Step36:  We can do boolean tests on lists as well
Step37:  Finally, note that you can also string multiple comparisons together, which can result in very intuitive tests
Step38:  If statements can have elif parts ("else if"), in addition to if/else parts. For example
Step39:  Of course we can combine if statements with for loops, to make a snippet that is almost interesting
Step40:  This is something of an advanced topic, but ordinary data types have boolean values associated with them, and, indeed, in early versions of Python there was not a separate boolean object. Essentially, anything that was a 0 value (the integer or floating point 0, an empty string "", or an empty list []) was False, and everything else was true. You can see the boolean value of any data object using the bool() function.
Step41:  Code Example
Step42:  Let's go through this line by line. First, we define the variable n, and set it to the integer 20. n is the length of the sequence we're going to form, and should probably have a better variable name. We then create a variable called sequence, and initialize it to the list with the integers 0 and 1 in it, the first two elements of the Fibonacci sequence. We have to create these elements "by hand", since the iterative part of the sequence requires two previous elements.
We then have a for loop over the list of integers from 2 (the next element of the list) to n (the length of the sequence). After the colon, we see a hash tag "#", and then a comment that if we had set n to some number less than 2 we would have a problem. Comments in Python start with #, and are good ways to make notes to yourself or to a user of your code explaining why you did what you did. Better than the comment here would be to test to make sure the value of n is valid, and to complain if it isn't; we'll try this later.
In the body of the loop, we append to the list an integer equal to the sum of the two previous elements of the list.
After exiting the loop (ending the indentation) we then print out the whole list. That's it!
Functions
We might want to use the Fibonacci snippet with different sequence lengths. We could cut an paste the code into another cell, changing the value of n, but it's easier and more useful to make a function out of the code. We do this with the def statement in Python
Step43:  We can now call fibonacci() for different sequence_lengths
Step44:  We've introduced a several new features here. First, note that the function itself is defined as a code block (a colon followed by an indented block). This is the standard way that Python delimits things. Next, note that the first line of the function is a single string. This is called a docstring, and is a special kind of comment that is often available to people using the function through the python command line
Step45:  If you define a docstring for all of your functions, it makes it easier for other people to use them, since they can get help on the arguments and return values of the function.
Next, note that rather than putting a comment in about what input values lead to errors, we have some testing of these values, followed by a warning if the value is invalid, and some conditional code to handle special cases.
Two More Data Structures
Step46:  Tuples are like lists, in that you can access the elements using indices
Step47:  However, tuples are immutable, you can't append to them or change the elements of them
Step48:  Tuples are useful anytime you want to group different pieces of data together in an object, but don't want to create a full-fledged class (see below) for them. For example, let's say you want the Cartesian coordinates of some objects in your program. Tuples are a good way to do this
Step49:  Again, it's not a necessary distinction, but one way to distinguish tuples and lists is that tuples are a collection of different things, here a name, and x and y coordinates, whereas a list is a collection of similar things, like if we wanted a list of those coordinates
Step50:  Tuples can be used when functions return more than one value. Say we wanted to compute the smallest x- and y-coordinates of the above list of objects. We could write
Step51:  Dictionaries are an object called "mappings" or "associative arrays" in other languages. Whereas a list associates an integer index with a set of objects
Step52:  The index in a dictionary is called the key, and the corresponding dictionary entry is the value. A dictionary can use (almost) anything as the key. Whereas lists are formed with square brackets [], dictionaries use curly brackets {}
Step53:  There's also a convenient way to create dictionaries without having to quote the keys.
Step54:  The len() command works on both tuples and dictionaries
Step55:  Conclusion of the Python Overview
There is, of course, much more to the language than I've covered here. I've tried to keep this brief enough so that you can jump in and start using Python to simplify your life and work. My own experience in learning new things is that the information doesn't "stick" unless you try and use it for something in real life.
You will no doubt need to learn more as you go. I've listed several other good references, including the Python Tutorial and Learn Python the Hard Way. Additionally, now is a good time to start familiarizing yourself with the Python Documentation, and, in particular, the Python Language Reference.
Tim Peters, one of the earliest and most prolific Python contributors, wrote the "Zen of Python", which can be accessed via the "import this" command
Step56:  No matter how experienced a programmer you are, these are words to meditate on.
II. Numpy and Scipy
Numpy contains core routines for doing fast vector, matrix, and linear algebra-type operations in Python. Scipy contains additional routines for optimization, special functions, and so on. Both contain modules written in C and Fortran so that they're as fast as possible. Together, they give Python roughly the same capability that the Matlab program offers. (In fact, if you're an experienced Matlab user, there a guide to Numpy for Matlab users just for you.)
Making vectors and matrices
Fundamental to both Numpy and Scipy is the ability to work with vectors and matrices. You can create vectors from lists using the array command
Step57:  size of the array
Step58:  To build matrices, you can either use the array command with lists of lists
Step59:  Add a column of ones to mat
Step60:  size of a matrix
Step61:  You can also form empty (zero) matrices of arbitrary shape (including vectors, which Numpy treats as vectors with one row), using the zeros command
Step62:  There's also an identity command that behaves as you'd expect
Step63:  as well as a ones command.
Linspace, matrix functions, and plotting
The linspace command makes a linear array of points from a starting to an ending value.
Step64:  If you provide a third argument, it takes that as the number of points in the space. If you don't provide the argument, it gives a length 50 linear space.
Step65:  linspace is an easy way to make coordinates for plotting. Functions in the numpy library (all of which are imported into IPython notebook) can act on an entire vector (or even a matrix) of points at once. Thus,
Step66:  In conjunction with matplotlib, this is a nice way to plot things
Step67:  Matrix operations
Matrix objects act sensibly when multiplied by scalars
Step68:  as well as when you add two matrices together. (However, the matrices have to be the same shape.)
Step69:  Something that confuses Matlab users is that the times (*) operator give element-wise multiplication rather than matrix multiplication
Step70:  To get matrix multiplication, you need the dot command
Step71:  dot can also do dot products (duh!)
Step72:  as well as matrix-vector products.
There are determinant, inverse, and transpose functions that act as you would suppose. Transpose can be abbreviated with ".T" at the end of a matrix object
Step73:  There's also a diag() function that takes a list or a vector and puts it along the diagonal of a square matrix.
Step75:  We'll find this useful later on.
Least squares fitting
Very often we deal with some data that we want to fit to some sort of expected behavior. Say we have the following
Step76:  There's a section below on parsing CSV data. We'll steal the parser from that. For an explanation, skip ahead to that section. Otherwise, just assume that this is a way to parse that text into a numpy array that we can plot and do other analyses with.
Step77:  Since we expect the data to have an exponential decay, we can plot it using a semi-log plot.
Step78:  For a pure exponential decay like this, we can fit the log of the data to a straight line. The above plot suggests this is a good approximation. Given a function
$$ y = Ae^{-ax} $$
$$ \log(y) = \log(A) - ax$$
Thus, if we fit the log of the data versus x, we should get a straight line with slope $a$, and an intercept that gives the constant $A$.
There's a numpy function called polyfit that will fit data to a polynomial form. We'll use this to fit to a straight line (a polynomial of order 1)
Step79:  Let's see whether this curve fits the data.
Step81:  If we have more complicated functions, we may not be able to get away with fitting to a simple polynomial. Consider the following data
Step82:  This data looks more Gaussian than exponential. If we wanted to, we could use polyfit for this as well, but let's use the curve_fit function from Scipy, which can fit to arbitrary functions. You can learn more using help(curve_fit).
First define a general Gaussian function to fit to.
Step83:  Now fit to it using curve_fit
Step84:  The curve_fit routine we just used is built on top of a very good general minimization capability in Scipy. You can learn more at the scipy documentation pages.
Monte Carlo and random numbers
Many methods in scientific computing rely on Monte Carlo integration, where a sequence of (pseudo) random numbers are used to approximate the integral of a function. Python has good random number generators in the standard library. The random() function gives pseudorandom numbers uniformly distributed between 0 and 1
Step85:  random() uses the Mersenne Twister algorithm, which is a highly regarded pseudorandom number generator. There are also functions to generate random integers, to randomly shuffle a list, and functions to pick random numbers from a particular distribution, like the normal distribution
Step86:  It is generally more efficient to generate a list of random numbers all at once, particularly if you're drawing from a non-uniform distribution. Numpy has functions to generate vectors and matrices of particular types of random distributions.
Step87:  III. Introduction to Pandas
Step88:  Series
A Series is a one-dimensional array-like object containing an array of data and an associated array of data labels.  The data can be any NumPy data type and the labels are the Series' index.
Create a Series
Step89:  Get the array representation of a Series
Step90:  Index objects are immutable and hold the axis labels and metadata such as names and axis names.
Get the index of the Series
Step91:  Create a Series with a custom index
Step92:  Get a value from a Series
Step93:  Get a set of values from a Series by passing in a list
Step94:  Get values great than 0
Step95:  Scalar multiply
Step96:  Apply a numpy math function
Step97:  A Series is like a fixed-length, ordered dict.  
Create a series by passing in a dict
Step98:  Re-order a Series by passing in an index (indices not found are NaN)
Step99:  Check for NaN with the pandas method
Step100:  Check for NaN with the Series method
Step101:  Series automatically aligns differently indexed data in arithmetic operations
Step102:  Name a Series
Step103:  Name a Series index
Step104:  Rename a Series' index in place
Step105:  DataFrame
A DataFrame is a tabular data structure containing an ordered collection of columns.  Each column can have a different type.  DataFrames have both row and column indices and is analogous to a dict of Series.  Row and column operations are treated roughly symmetrically.  Columns returned when indexing a DataFrame are views of the underlying data, not a copy.  To obtain a copy, use the Series' copy method.
Create a DataFrame
Step106:  Like Series, columns that are not present in the data are NaN
Step107:  Retrieve a column by key, returning a Series
Step108:  Retrive a column by attribute, returning a Series
Step109:  Retrieve a row by position
Step110:  Update a column by assignment
Step111:  Assign a Series to a column (note if assigning a list or array, the length must match the DataFrame, unlike a Series)
Step112:  Assign a new column that doesn't exist to create a new column
Step113:  Delete a column
Step114:  Transpose the DataFrame
Step115:  Create a DataFrame from a nested dict of dicts (the keys in the inner dicts are unioned and sorted to form the index in the result, unless an explicit index is specified)
Step116:  Create a DataFrame from a dict of Series
Step117:  Set the DataFrame index name
Step118:  Set the DataFrame columns name
Step119:  Return the data contained in a DataFrame as a 2D ndarray
Step120:  If the columns are different dtypes, the 2D ndarray's dtype will accomodate all of the columns
Step121:  Reindexing
Create a new object with the data conformed to a new index.  Any missing values are set to NaN.
Step122:  Reindexing rows returns a new frame with the specified index
Step123:  Reindex columns
Step124:  Dropping Entries
Drop rows from a Series or DataFrame
Step125:  Indexing, Selecting, Filtering
Pandas supports indexing into a DataFrame.
Step126:  Select specified columns from a DataFrame
Step127:  Select a slice from a DataFrame
Step128:  Select from a DataFrame based on a filter
Step129:  Select a slice of rows from a specific column of a DataFrame
Step130:  Arithmetic and Data Alignment
Adding DataFrame objects results in the union of index pairs for rows and columns if the pairs are not the same, resulting in NaN for indices that do not overlap
Step131:  Set a fill value instead of NaN for indices that do not overlap
Step132:  Like NumPy, pandas supports arithmetic operations between DataFrames and Series.
Match the index of the Series on the DataFrame's columns, broadcasting down the rows
Step133:  Match the index of the Series on the DataFrame's columns, broadcasting down the rows and union the indices that do not match
Step134:  Function Application and Mapping
NumPy ufuncs (element-wise array methods) operate on pandas objects
Step135:  Apply a function on 1D arrays to each column
Step136:  Apply a function on 1D arrays to each row
Step137:  Apply an element-wise Python function to a DataFrame
Step138:  Sorting
Step139:  Sort a DataFrame by its index
Step140:  Sort a DataFrame by columns in descending order
Step141:  Sort a DataFrame's values by column
Step142:  Summarizing and Computing Descriptive Statistics
Unlike NumPy arrays, Pandas descriptive statistics automatically exclude missing data.  NaN values are excluded unless the entire row or column is NA.
Step143:  Sum and Mean
Step144:  Descriptive analysis
Step145:  Pivot tables
group by cat1 and calculate mean
<ASSISTANT_TASK:>
Python Code:
import sys
print('Python version:', sys.version)
import IPython
print('IPython:', IPython.__version__)
import numpy
print('numpy:', numpy.__version__)
import scipy
print('scipy:', scipy.__version__)
import matplotlib
print('matplotlib:', matplotlib.__version__)
import pandas
print('pandas:', pandas.__version__)
import sklearn
print('scikit-learn:', sklearn.__version__)
Explanation: 02 - Introduction to Python for Data Analysis
by Alejandro Correa Bahnsen and Jesus Solano
version 1.4, January 2019
Part of the class Practical Machine Learning
This notebook is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. Special thanks goes to Rick Muller, Sandia National Laboratories
Why Python?
Python is the programming language of choice for many scientists to a large degree because it offers a great deal of power to analyze and model scientific data with relatively little overhead in terms of learning, installation or development time. It is a language you can pick up in a weekend, and use for the rest of one's life.
The Python Tutorial is a great place to start getting a feel for the language. To complement this material, I taught a Python Short Course years ago to a group of computational chemists during a time that I was worried the field was moving too much in the direction of using canned software rather than developing one's own methods. I wanted to focus on what working scientists needed to be more productive: parsing output of other programs, building simple models, experimenting with object oriented programming, extending the language with C, and simple GUIs. 
I'm trying to do something very similar here, to cut to the chase and focus on what scientists need. In the last year or so, the Jupyter Project has put together a notebook interface that I have found incredibly valuable. A large number of people have released very good IPython Notebooks that I have taken a huge amount of pleasure reading through. Some ones that I particularly like include:
Rick Muller A Crash Course in Python for Scientists
Rob Johansson's excellent notebooks, including Scientific Computing with Python and Computational Quantum Physics with QuTiP lectures;
XKCD style graphs in matplotlib;
A collection of Notebooks for using IPython effectively
A gallery of interesting IPython Notebooks
I find Jupyter notebooks an easy way both to get important work done in my everyday job, as well as to communicate what I've done, how I've done it, and why it matters to my coworkers. In the interest of putting more notebooks out into the wild for other people to use and enjoy, I thought I would try to recreate some of what I was trying to get across in the original Python Short Course, updated by 15 years of Python, Numpy, Scipy, Pandas, Matplotlib, and IPython development, as well as my own experience in using Python almost every day of this time.
Why Python for Data Analysis?
Python is great for scripting and applications.
The pandas library offers improved library support.
Scraping, web APIs
Strong High Performance Computation support
Load balanceing tasks
MPI, GPU
MapReduce
Strong support for abstraction
Intel MKL
HDF5
Environment
But we already know R
...Which is better? Hard to answer
http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html
http://www.kdnuggets.com/2015/03/the-grammar-data-science-python-vs-r.html
https://www.datacamp.com/community/tutorials/r-or-python-for-data-analysis
https://www.dataquest.io/blog/python-vs-r/
http://www.dataschool.io/python-or-r-for-data-science/
What You Need to Install
There are two branches of current releases in Python: the older-syntax Python 2, and the newer-syntax Python 3. This schizophrenia is largely intentional: when it became clear that some non-backwards-compatible changes to the language were necessary, the Python dev-team decided to go through a five-year (or so) transition, during which the new language features would be introduced and the old language was still actively maintained, to make such a transition as easy as possible. 
Nonetheless, I'm going to write these notes with Python 3 in mind, since this is the version of the language that I use in my day-to-day job, and am most comfortable with. 
With this in mind, these notes assume you have a Python distribution that includes:
Python version 3.5;
Numpy, the core numerical extensions for linear algebra and multidimensional arrays;
Scipy, additional libraries for scientific programming;
Matplotlib, excellent plotting and graphing libraries;
IPython, with the additional libraries required for the notebook interface.
Pandas, Python version of R dataframe
scikit-learn, Machine learning library!
A good, easy to install option that supports Mac, Windows, and Linux, and that has all of these packages (and much more) is the Anaconda.
Checking your installation
You can run the following code to check the versions of the packages on your system:
(in IPython notebook, press shift and return together to execute the contents of a cell)
End of explanation
2+2
(50-5*6)/4
Explanation: I. Python Overview
This is a quick introduction to Python. There are lots of other places to learn the language more thoroughly. I have collected a list of useful links, including ones to other learning resources, at the end of this notebook. If you want a little more depth, Python Tutorial is a great place to start, as is Zed Shaw's Learn Python the Hard Way.
The lessons that follow make use of the IPython notebooks. There's a good introduction to notebooks in the IPython notebook documentation that even has a nice video on how to use the notebooks. You should probably also flip through the IPython tutorial in your copious free time.
Briefly, notebooks have code cells (that are generally followed by result cells) and text cells. The text cells are the stuff that you're reading now. The code cells start with "In []:" with some number generally in the brackets. If you put your cursor in the code cell and hit Shift-Enter, the code will run in the Python interpreter and the result will print out in the output cell. You can then change things around and see whether you understand what's going on. If you need to know more, see the IPython notebook documentation or the IPython tutorial.
Using Python as a Calculator
Many of the things I used to use a calculator for, I now use Python for:
End of explanation
sqrt(81)
from math import sqrt
sqrt(81)
Explanation: (If you're typing this into an IPython notebook, or otherwise using notebook file, you hit shift-Enter to evaluate a cell.)
In the last few lines, we have sped by a lot of things that we should stop for a moment and explore a little more fully. We've seen, however briefly, two different data types: integers, also known as whole numbers to the non-programming world, and floating point numbers, also known (incorrectly) as decimal numbers to the rest of the world.
We've also seen the first instance of an import statement. Python has a huge number of libraries included with the distribution. To keep things simple, most of these variables and functions are not accessible from a normal Python interactive session. Instead, you have to import the name. For example, there is a math module containing many useful functions. To access, say, the square root function, you can either first
from math import sqrt
and then
End of explanation
import math
math.sqrt(81)
Explanation: or you can simply import the math library itself
End of explanation
radius = 20
pi = math.pi
area = pi * radius ** 2 
area
Explanation: You can define variables using the equals (=) sign:
End of explanation
return = 0
Explanation: You can name a variable almost anything you want. It needs to start with an alphabetical character or "_", can contain alphanumeric characters plus underscores ("_"). Certain words, however, are reserved for the language:
and, as, assert, break, class, continue, def, del, elif, else, except, 
exec, finally, for, from, global, if, import, in, is, lambda, not, or,
pass, print, raise, return, try, while, with, yield
Trying to define a variable using one of these will result in a syntax error:
End of explanation
'Hello, World!'
Explanation: The Python Tutorial has more on using Python as an interactive shell. The IPython tutorial makes a nice complement to this, since IPython has a much more sophisticated iteractive shell.
Strings
Strings are lists of printable characters, and can be defined using either single quotes
End of explanation
"Hello, World!"
Explanation: or double quotes
End of explanation
greeting = "Hello, World!"
Explanation: Just like the other two data objects we're familiar with (ints and floats), you can assign a string to a variable
End of explanation
print(greeting)
Explanation: The print statement is often used for printing character strings:
End of explanation
print("The area is " + area)
print("The area is " + str(area))
Explanation: But it can also print data types other than strings:
End of explanation
statement = "Hello, " + "World!"
print(statement)
Explanation: In the above snipped, the number 600 (stored in the variable "area") is converted into a string before being printed out.
You can use the + operator to concatenate strings together:
Don't forget the space between the strings, if you want one there.
End of explanation
days_of_the_week = ["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"]
Explanation: If you have a lot of words to concatenate together, there are other, more efficient ways to do this. But this is fine for linking a few strings together.
Lists
Very often in a programming language, one wants to keep a group of similar items together. Python does this using a data type called lists.
End of explanation
days_of_the_week[2]
Explanation: You can access members of the list using the index of that item:
End of explanation
days_of_the_week[-1]
Explanation: Python lists, like C, but unlike Fortran, use 0 as the index of the first element of a list. Thus, in this example, the 0 element is "Sunday", 1 is "Monday", and so on. If you need to access the nth element from the end of the list, you can use a negative index. For example, the -1 element of a list is the last element:
End of explanation
languages = ["Fortran","C","C++"]
languages.append("Python")
print(languages)
Explanation: You can add additional items to the list using the .append() command:
End of explanation
list(range(10))
Explanation: The range() command is a convenient way to make sequential lists of numbers:
End of explanation
list(range(2,8))
Explanation: Note that range(n) starts at 0 and gives the sequential list of integers less than n. If you want to start at a different number, use range(start,stop)
End of explanation
evens = list(range(0,20,2))
evens
evens[3]
Explanation: The lists created above with range have a step of 1 between elements. You can also give a fixed step size via a third command:
End of explanation
["Today",7,99.3,""]
Explanation: Lists do not have to hold the same data type. For example,
End of explanation
help(len)
len(evens)
Explanation: However, it's good (but not essential) to use lists for similar objects that are somehow logically connected. If you want to group different data types together into a composite data object, it's best to use tuples, which we will learn about below.
You can find out how long a list is using the len() command:
End of explanation
for day in days_of_the_week:
    print(day)
Explanation: Iteration, Indentation, and Blocks
One of the most useful things you can do with lists is to iterate through them, i.e. to go through each element one at a time. To do this in Python, we use the for statement:
End of explanation
for day in days_of_the_week:
    statement = "Today is " + day
    print(statement)
Explanation: This code snippet goes through each element of the list called days_of_the_week and assigns it to the variable day. It then executes everything in the indented block (in this case only one line of code, the print statement) using those variable assignments. When the program has gone through every element of the list, it exists the block.
(Almost) every programming language defines blocks of code in some way. In Fortran, one uses END statements (ENDDO, ENDIF, etc.) to define code blocks. In C, C++, and Perl, one uses curly braces {} to define these blocks.
Python uses a colon (":"), followed by indentation level to define code blocks. Everything at a higher level of indentation is taken to be in the same block. In the above example the block was only a single line, but we could have had longer blocks as well:
End of explanation
for i in range(20):
    print("The square of ",i," is ",i*i)
Explanation: The range() command is particularly useful with the for statement to execute loops of a specified length:
End of explanation
for letter in "Sunday":
    print(letter)
Explanation: Slicing
Lists and strings have something in common that you might not suspect: they can both be treated as sequences. You already know that you can iterate through the elements of a list. You can also iterate through the letters in a string:
End of explanation
days_of_the_week[0]
Explanation: This is only occasionally useful. Slightly more useful is the slicing operation, which you can also use on any sequence. We already know that we can use indexing to get the first element of a list:
End of explanation
days_of_the_week[0:2]
Explanation: If we want the list containing the first two elements of a list, we can do this via
End of explanation
days_of_the_week[:2]
Explanation: or simply
End of explanation
days_of_the_week[-2:]
Explanation: If we want the last items of the list, we can do this with negative slicing:
End of explanation
workdays = days_of_the_week[1:6]
print(workdays)
Explanation: which is somewhat logically consistent with negative indices accessing the last elements of the list.
You can do:
End of explanation
day = "Sunday"
abbreviation = day[:3]
print(abbreviation)
Explanation: Since strings are sequences, you can also do this to them:
End of explanation
numbers = list(range(0,40))
evens = numbers[2::2]
evens
Explanation: If we really want to get fancy, we can pass a third element into the slice, which specifies a step length (just like a third argument to the range() function specifies the step):
End of explanation
if day == "Sunday":
    print("Sleep in")
else:
    print("Go to work")
Explanation: Note that in this example I was even able to omit the second argument, so that the slice started at 2, went to the end of the list, and took every second element, to generate the list of even numbers less that 40.
Booleans and Truth Testing
We have now learned a few data types. We have integers and floating point numbers, strings, and lists to contain them. We have also learned about lists, a container that can hold any data type. We have learned to print things out, and to iterate over items in lists. We will now learn about boolean variables that can be either True or False.
We invariably need some concept of conditions in programming to control branching behavior, to allow a program to react differently to different situations. If it's Monday, I'll go to work, but if it's Sunday, I'll sleep in. To do this in Python, we use a combination of boolean variables, which evaluate to either True or False, and if statements, that control branching based on boolean values.
For example:
End of explanation
day == "Sunday"
Explanation: (Quick quiz: why did the snippet print "Go to work" here? What is the variable "day" set to?)
Let's take the snippet apart to see what happened. First, note the statement
End of explanation
1 == 2
50 == 2*25
3 < 3.14159
1 == 1.0
1 != 0
1 <= 2
1 >= 1
Explanation: If we evaluate it by itself, as we just did, we see that it returns a boolean value, False. The "==" operator performs equality testing. If the two items are equal, it returns True, otherwise it returns False. In this case, it is comparing two variables, the string "Sunday", and whatever is stored in the variable "day", which, in this case, is the other string "Saturday". Since the two strings are not equal to each other, the truth test has the false value.
The if statement that contains the truth test is followed by a code block (a colon followed by an indented block of code). If the boolean is true, it executes the code in that block. Since it is false in the above example, we don't see that code executed.
The first block of code is followed by an else statement, which is executed if nothing else in the above if statement is true. Since the value was false, this code is executed, which is why we see "Go to work".
You can compare any data types in Python:
End of explanation
1 is 1.0
Explanation: We see a few other boolean operators here, all of which which should be self-explanatory. Less than, equality, non-equality, and so on.
Particularly interesting is the 1 == 1.0 test, which is true, since even though the two objects are different data types (integer and floating point number), they have the same value. There is another boolean operator is, that tests whether two objects are the same object:
End of explanation
[1,2,3] == [1,2,4]
[1,2,3] < [1,2,4]
Explanation: We can do boolean tests on lists as well:
End of explanation
hours = 5
0 < hours < 24
Explanation: Finally, note that you can also string multiple comparisons together, which can result in very intuitive tests:
End of explanation
if day == "Sunday":
    print("Sleep in")
elif day == "Saturday":
    print("Do chores")
else:
    print("Go to work")
Explanation: If statements can have elif parts ("else if"), in addition to if/else parts. For example:
End of explanation
for day in days_of_the_week:
    statement = "Today is " + day
    print(statement)
    if day == "Sunday":
        print("   Sleep in")
    elif day == "Saturday":
        print("   Do chores")
    else:
        print("   Go to work")
Explanation: Of course we can combine if statements with for loops, to make a snippet that is almost interesting:
End of explanation
bool(1)
bool(0)
bool(["This "," is "," a "," list"])
Explanation: This is something of an advanced topic, but ordinary data types have boolean values associated with them, and, indeed, in early versions of Python there was not a separate boolean object. Essentially, anything that was a 0 value (the integer or floating point 0, an empty string "", or an empty list []) was False, and everything else was true. You can see the boolean value of any data object using the bool() function.
End of explanation
n = 10
sequence = [0,1]
for i in range(2,n): # This is going to be a problem if we ever set n <= 2!
    sequence.append(sequence[i-1]+sequence[i-2])
print(sequence)
Explanation: Code Example: The Fibonacci Sequence
The Fibonacci sequence is a sequence in math that starts with 0 and 1, and then each successive entry is the sum of the previous two. Thus, the sequence goes 0,1,1,2,3,5,8,13,21,34,55,89,...
A very common exercise in programming books is to compute the Fibonacci sequence up to some number n. First I'll show the code, then I'll discuss what it is doing.
End of explanation
def fibonacci(sequence_length):
    "Return the Fibonacci sequence of length *sequence_length*"
    sequence = [0,1]
    if sequence_length < 1:
        print("Fibonacci sequence only defined for length 1 or greater")
        return
    if 0 < sequence_length < 3:
        return sequence[:sequence_length]
    for i in range(2,sequence_length): 
        sequence.append(sequence[i-1]+sequence[i-2])
    return sequence
Explanation: Let's go through this line by line. First, we define the variable n, and set it to the integer 20. n is the length of the sequence we're going to form, and should probably have a better variable name. We then create a variable called sequence, and initialize it to the list with the integers 0 and 1 in it, the first two elements of the Fibonacci sequence. We have to create these elements "by hand", since the iterative part of the sequence requires two previous elements.
We then have a for loop over the list of integers from 2 (the next element of the list) to n (the length of the sequence). After the colon, we see a hash tag "#", and then a comment that if we had set n to some number less than 2 we would have a problem. Comments in Python start with #, and are good ways to make notes to yourself or to a user of your code explaining why you did what you did. Better than the comment here would be to test to make sure the value of n is valid, and to complain if it isn't; we'll try this later.
In the body of the loop, we append to the list an integer equal to the sum of the two previous elements of the list.
After exiting the loop (ending the indentation) we then print out the whole list. That's it!
Functions
We might want to use the Fibonacci snippet with different sequence lengths. We could cut an paste the code into another cell, changing the value of n, but it's easier and more useful to make a function out of the code. We do this with the def statement in Python:
End of explanation
fibonacci(2)
fibonacci(12)
Explanation: We can now call fibonacci() for different sequence_lengths:
End of explanation
help(fibonacci)
Explanation: We've introduced a several new features here. First, note that the function itself is defined as a code block (a colon followed by an indented block). This is the standard way that Python delimits things. Next, note that the first line of the function is a single string. This is called a docstring, and is a special kind of comment that is often available to people using the function through the python command line:
End of explanation
t = (1,2,'hi',9.0)
t
Explanation: If you define a docstring for all of your functions, it makes it easier for other people to use them, since they can get help on the arguments and return values of the function.
Next, note that rather than putting a comment in about what input values lead to errors, we have some testing of these values, followed by a warning if the value is invalid, and some conditional code to handle special cases.
Two More Data Structures: Tuples and Dictionaries
Before we end the Python overview, I wanted to touch on two more data structures that are very useful (and thus very common) in Python programs.
A tuple is a sequence object like a list or a string. It's constructed by grouping a sequence of objects together with commas, either without brackets, or with parentheses:
End of explanation
t[1]
Explanation: Tuples are like lists, in that you can access the elements using indices:
End of explanation
t.append(7)
t[1]=77
Explanation: However, tuples are immutable, you can't append to them or change the elements of them:
End of explanation
('Bob',0.0,21.0)
Explanation: Tuples are useful anytime you want to group different pieces of data together in an object, but don't want to create a full-fledged class (see below) for them. For example, let's say you want the Cartesian coordinates of some objects in your program. Tuples are a good way to do this:
End of explanation
positions = [
             ('Bob',0.0,21.0),
             ('Cat',2.5,13.1),
             ('Dog',33.0,1.2)
             ]
Explanation: Again, it's not a necessary distinction, but one way to distinguish tuples and lists is that tuples are a collection of different things, here a name, and x and y coordinates, whereas a list is a collection of similar things, like if we wanted a list of those coordinates:
End of explanation
def minmax(objects):
    minx = 1e20 # These are set to really big numbers
    miny = 1e20
    for obj in objects:
        name,x,y = obj
        if x < minx: 
            minx = x
        if y < miny:
            miny = y
    return minx,miny
x,y = minmax(positions)
print(x,y)
Explanation: Tuples can be used when functions return more than one value. Say we wanted to compute the smallest x- and y-coordinates of the above list of objects. We could write:
End of explanation
mylist = [1,2,9,21]
Explanation: Dictionaries are an object called "mappings" or "associative arrays" in other languages. Whereas a list associates an integer index with a set of objects:
End of explanation
ages = {"Rick": 46, "Bob": 86, "Fred": 21}
print("Rick's age is ",ages["Rick"])
Explanation: The index in a dictionary is called the key, and the corresponding dictionary entry is the value. A dictionary can use (almost) anything as the key. Whereas lists are formed with square brackets [], dictionaries use curly brackets {}:
End of explanation
dict(Rick=46,Bob=86,Fred=20)
Explanation: There's also a convenient way to create dictionaries without having to quote the keys.
End of explanation
len(t)
len(ages)
Explanation: The len() command works on both tuples and dictionaries:
End of explanation
import this
Explanation: Conclusion of the Python Overview
There is, of course, much more to the language than I've covered here. I've tried to keep this brief enough so that you can jump in and start using Python to simplify your life and work. My own experience in learning new things is that the information doesn't "stick" unless you try and use it for something in real life.
You will no doubt need to learn more as you go. I've listed several other good references, including the Python Tutorial and Learn Python the Hard Way. Additionally, now is a good time to start familiarizing yourself with the Python Documentation, and, in particular, the Python Language Reference.
Tim Peters, one of the earliest and most prolific Python contributors, wrote the "Zen of Python", which can be accessed via the "import this" command:
End of explanation
import numpy as np
import scipy as sp
array = np.array([1,2,3,4,5,6])
array
Explanation: No matter how experienced a programmer you are, these are words to meditate on.
II. Numpy and Scipy
Numpy contains core routines for doing fast vector, matrix, and linear algebra-type operations in Python. Scipy contains additional routines for optimization, special functions, and so on. Both contain modules written in C and Fortran so that they're as fast as possible. Together, they give Python roughly the same capability that the Matlab program offers. (In fact, if you're an experienced Matlab user, there a guide to Numpy for Matlab users just for you.)
Making vectors and matrices
Fundamental to both Numpy and Scipy is the ability to work with vectors and matrices. You can create vectors from lists using the array command:
End of explanation
array.shape
Explanation: size of the array
End of explanation
mat = np.array([[0,1],[1,0]])
mat
Explanation: To build matrices, you can either use the array command with lists of lists:
End of explanation
mat2 = np.c_[mat, np.ones(2)]
mat2
Explanation: Add a column of ones to mat
End of explanation
mat2.shape
Explanation: size of a matrix
End of explanation
np.zeros((3,3))
Explanation: You can also form empty (zero) matrices of arbitrary shape (including vectors, which Numpy treats as vectors with one row), using the zeros command:
End of explanation
np.identity(4)
Explanation: There's also an identity command that behaves as you'd expect:
End of explanation
np.linspace(0,1)
Explanation: as well as a ones command.
Linspace, matrix functions, and plotting
The linspace command makes a linear array of points from a starting to an ending value.
End of explanation
np.linspace(0,1,11)
Explanation: If you provide a third argument, it takes that as the number of points in the space. If you don't provide the argument, it gives a length 50 linear space.
End of explanation
x = np.linspace(0,2*np.pi)
np.sin(x)
Explanation: linspace is an easy way to make coordinates for plotting. Functions in the numpy library (all of which are imported into IPython notebook) can act on an entire vector (or even a matrix) of points at once. Thus,
End of explanation
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('ggplot')
plt.plot(x,np.sin(x))
Explanation: In conjunction with matplotlib, this is a nice way to plot things:
End of explanation
0.125*np.identity(3)
Explanation: Matrix operations
Matrix objects act sensibly when multiplied by scalars:
End of explanation
np.identity(2) + np.array([[1,1],[1,2]])
Explanation: as well as when you add two matrices together. (However, the matrices have to be the same shape.)
End of explanation
np.identity(2)*np.ones((2,2))
Explanation: Something that confuses Matlab users is that the times (*) operator give element-wise multiplication rather than matrix multiplication:
End of explanation
np.dot(np.identity(2),np.ones((2,2)))
Explanation: To get matrix multiplication, you need the dot command:
End of explanation
v = np.array([3,4])
np.sqrt(np.dot(v,v))
Explanation: dot can also do dot products (duh!):
End of explanation
m = np.array([[1,2],[3,4]])
m.T
np.linalg.inv(m)
Explanation: as well as matrix-vector products.
There are determinant, inverse, and transpose functions that act as you would suppose. Transpose can be abbreviated with ".T" at the end of a matrix object:
End of explanation
np.diag([1,2,3,4,5])
Explanation: There's also a diag() function that takes a list or a vector and puts it along the diagonal of a square matrix.
End of explanation
raw_data = \
3.1905781584582433,0.028208609537968457
4.346895074946466,0.007160804747670053
5.374732334047101,0.0046962988461934805
8.201284796573875,0.0004614473299618756
10.899357601713055,0.00005038370219939726
16.295503211991434,4.377451812785309e-7
21.82012847965739,3.0799922117601088e-9
32.48394004282656,1.524776208284536e-13
43.53319057815846,5.5012073588707224e-18
Explanation: We'll find this useful later on.
Least squares fitting
Very often we deal with some data that we want to fit to some sort of expected behavior. Say we have the following:
End of explanation
data = []
for line in raw_data.splitlines():
    words = line.split(',')
    data.append(words)
data = np.array(data, dtype=np.float)
data
data[:, 0]
plt.title("Raw Data")
plt.xlabel("Distance")
plt.plot(data[:,0],data[:,1],'bo')
Explanation: There's a section below on parsing CSV data. We'll steal the parser from that. For an explanation, skip ahead to that section. Otherwise, just assume that this is a way to parse that text into a numpy array that we can plot and do other analyses with.
End of explanation
plt.title("Raw Data")
plt.xlabel("Distance")
plt.semilogy(data[:,0],data[:,1],'bo')
Explanation: Since we expect the data to have an exponential decay, we can plot it using a semi-log plot.
End of explanation
params = sp.polyfit(data[:,0],np.log(data[:,1]),1)
a = params[0]
A = np.exp(params[1])
Explanation: For a pure exponential decay like this, we can fit the log of the data to a straight line. The above plot suggests this is a good approximation. Given a function
$$ y = Ae^{-ax} $$
$$ \log(y) = \log(A) - ax$$
Thus, if we fit the log of the data versus x, we should get a straight line with slope $a$, and an intercept that gives the constant $A$.
There's a numpy function called polyfit that will fit data to a polynomial form. We'll use this to fit to a straight line (a polynomial of order 1)
End of explanation
x = np.linspace(1,45)
plt.title("Raw Data")
plt.xlabel("Distance")
plt.semilogy(data[:,0],data[:,1],'bo')
plt.semilogy(x,A*np.exp(a*x),'b-')
Explanation: Let's see whether this curve fits the data.
End of explanation
gauss_data = \
-0.9902286902286903,1.4065274110372852e-19
-0.7566104566104566,2.2504438576596563e-18
-0.5117810117810118,1.9459459459459454
-0.31887271887271884,10.621621621621626
-0.250997150997151,15.891891891891893
-0.1463309463309464,23.756756756756754
-0.07267267267267263,28.135135135135133
-0.04426734426734419,29.02702702702703
-0.0015939015939017698,29.675675675675677
0.04689304689304685,29.10810810810811
0.0840994840994842,27.324324324324326
0.1700546700546699,22.216216216216214
0.370878570878571,7.540540540540545
0.5338338338338338,1.621621621621618
0.722014322014322,0.08108108108108068
0.9926849926849926,-0.08108108108108646
data = []
for line in gauss_data.splitlines():
    words = line.split(',')
    data.append(words)
data = np.array(data, dtype=np.float)
plt.plot(data[:,0],data[:,1],'bo')
Explanation: If we have more complicated functions, we may not be able to get away with fitting to a simple polynomial. Consider the following data:
End of explanation
def gauss(x,A,a): 
    return A*np.exp(a*x**2)
Explanation: This data looks more Gaussian than exponential. If we wanted to, we could use polyfit for this as well, but let's use the curve_fit function from Scipy, which can fit to arbitrary functions. You can learn more using help(curve_fit).
First define a general Gaussian function to fit to.
End of explanation
from scipy.optimize import curve_fit
params,conv = curve_fit(gauss,data[:,0],data[:,1])
x = np.linspace(-1,1)
plt.plot(data[:,0],data[:,1],'bo')
A,a = params
plt.plot(x,gauss(x,A,a),'g-')
Explanation: Now fit to it using curve_fit:
End of explanation
from random import random
rands = []
for i in range(100):
    rands.append(random())
plt.plot(rands)
Explanation: The curve_fit routine we just used is built on top of a very good general minimization capability in Scipy. You can learn more at the scipy documentation pages.
Monte Carlo and random numbers
Many methods in scientific computing rely on Monte Carlo integration, where a sequence of (pseudo) random numbers are used to approximate the integral of a function. Python has good random number generators in the standard library. The random() function gives pseudorandom numbers uniformly distributed between 0 and 1:
End of explanation
from random import gauss
grands = []
for i in range(100):
    grands.append(gauss(0,1))
plt.plot(grands)
Explanation: random() uses the Mersenne Twister algorithm, which is a highly regarded pseudorandom number generator. There are also functions to generate random integers, to randomly shuffle a list, and functions to pick random numbers from a particular distribution, like the normal distribution:
End of explanation
plt.plot(np.random.rand(100))
Explanation: It is generally more efficient to generate a list of random numbers all at once, particularly if you're drawing from a non-uniform distribution. Numpy has functions to generate vectors and matrices of particular types of random distributions.
End of explanation
import pandas as pd
import numpy as np
Explanation: III. Introduction to Pandas
End of explanation
ser_1 = pd.Series([1, 1, 2, -3, -5, 8, 13])
ser_1
Explanation: Series
A Series is a one-dimensional array-like object containing an array of data and an associated array of data labels.  The data can be any NumPy data type and the labels are the Series' index.
Create a Series:
End of explanation
ser_1.values
Explanation: Get the array representation of a Series:
End of explanation
ser_1.index
Explanation: Index objects are immutable and hold the axis labels and metadata such as names and axis names.
Get the index of the Series:
End of explanation
ser_2 = pd.Series([1, 1, 2, -3, -5], index=['a', 'b', 'c', 'd', 'e'])
ser_2
Explanation: Create a Series with a custom index:
End of explanation
ser_2[4] == ser_2['e']
Explanation: Get a value from a Series:
End of explanation
ser_2[['c', 'a', 'b']]
Explanation: Get a set of values from a Series by passing in a list:
End of explanation
ser_2[ser_2 > 0]
Explanation: Get values great than 0:
End of explanation
ser_2 * 2
Explanation: Scalar multiply:
End of explanation
np.exp(ser_2)
Explanation: Apply a numpy math function:
End of explanation
dict_1 = {'foo' : 100, 'bar' : 200, 'baz' : 300}
ser_3 = pd.Series(dict_1)
ser_3
Explanation: A Series is like a fixed-length, ordered dict.  
Create a series by passing in a dict:
End of explanation
index = ['foo', 'bar', 'baz', 'qux']
ser_4 = pd.Series(dict_1, index=index)
ser_4
Explanation: Re-order a Series by passing in an index (indices not found are NaN):
End of explanation
pd.isnull(ser_4)
Explanation: Check for NaN with the pandas method:
End of explanation
ser_4.isnull()
Explanation: Check for NaN with the Series method:
End of explanation
ser_3 + ser_4
Explanation: Series automatically aligns differently indexed data in arithmetic operations:
End of explanation
ser_4.name = 'foobarbazqux'
Explanation: Name a Series:
End of explanation
ser_4.index.name = 'label'
ser_4
Explanation: Name a Series index:
End of explanation
ser_4.index = ['fo', 'br', 'bz', 'qx']
ser_4
Explanation: Rename a Series' index in place:
End of explanation
data_1 = {'state' : ['VA', 'VA', 'VA', 'MD', 'MD'],
          'year' : [2012, 2013, 2014, 2014, 2015],
          'pop' : [5.0, 5.1, 5.2, 4.0, 4.1]}
df_1 = pd.DataFrame(data_1)
df_1
df_2 = pd.DataFrame(data_1, columns=['year', 'state', 'pop'])
df_2
Explanation: DataFrame
A DataFrame is a tabular data structure containing an ordered collection of columns.  Each column can have a different type.  DataFrames have both row and column indices and is analogous to a dict of Series.  Row and column operations are treated roughly symmetrically.  Columns returned when indexing a DataFrame are views of the underlying data, not a copy.  To obtain a copy, use the Series' copy method.
Create a DataFrame:
End of explanation
df_3 = pd.DataFrame(data_1, columns=['year', 'state', 'pop', 'unempl'])
df_3
Explanation: Like Series, columns that are not present in the data are NaN:
End of explanation
df_3['state']
Explanation: Retrieve a column by key, returning a Series:
End of explanation
df_3.year
Explanation: Retrive a column by attribute, returning a Series:
End of explanation
df_3.iloc[0]
Explanation: Retrieve a row by position:
End of explanation
df_3['unempl'] = np.arange(5)
df_3
Explanation: Update a column by assignment:
End of explanation
unempl = pd.Series([6.0, 6.0, 6.1], index=[2, 3, 4])
df_3['unempl'] = unempl
df_3
Explanation: Assign a Series to a column (note if assigning a list or array, the length must match the DataFrame, unlike a Series):
End of explanation
df_3['state_dup'] = df_3['state']
df_3
Explanation: Assign a new column that doesn't exist to create a new column:
End of explanation
del df_3['state_dup']
df_3
Explanation: Delete a column:
End of explanation
df_3.T
Explanation: Transpose the DataFrame:
End of explanation
pop = {'VA' : {2013 : 5.1, 2014 : 5.2},
       'MD' : {2014 : 4.0, 2015 : 4.1}}
df_4 = pd.DataFrame(pop)
df_4
Explanation: Create a DataFrame from a nested dict of dicts (the keys in the inner dicts are unioned and sorted to form the index in the result, unless an explicit index is specified):
End of explanation
data_2 = {'VA' : df_4['VA'][1:],
          'MD' : df_4['MD'][2:]}
df_5 = pd.DataFrame(data_2)
df_5
Explanation: Create a DataFrame from a dict of Series:
End of explanation
df_5.index.name = 'year'
df_5
Explanation: Set the DataFrame index name:
End of explanation
df_5.columns.name = 'state'
df_5
Explanation: Set the DataFrame columns name:
End of explanation
df_5.values
Explanation: Return the data contained in a DataFrame as a 2D ndarray:
End of explanation
df_3.values
Explanation: If the columns are different dtypes, the 2D ndarray's dtype will accomodate all of the columns:
End of explanation
df_3
Explanation: Reindexing
Create a new object with the data conformed to a new index.  Any missing values are set to NaN.
End of explanation
df_3.reindex(list(reversed(range(0, 6))))
Explanation: Reindexing rows returns a new frame with the specified index:
End of explanation
df_3.reindex(columns=['state', 'pop', 'unempl', 'year'])
Explanation: Reindex columns:
End of explanation
df_7 = df_3.drop([0, 1])
df_7
df_7 = df_7.drop('unempl', axis=1)
df_7
Explanation: Dropping Entries
Drop rows from a Series or DataFrame:
End of explanation
df_3
Explanation: Indexing, Selecting, Filtering
Pandas supports indexing into a DataFrame.
End of explanation
df_3[['pop', 'unempl']]
Explanation: Select specified columns from a DataFrame:
End of explanation
df_3[:2]
df_3.iloc[1:3]
Explanation: Select a slice from a DataFrame:
End of explanation
df_3[df_3['pop'] > 5]
Explanation: Select from a DataFrame based on a filter:
End of explanation
df_3.loc[0:2, 'pop']
df_3
Explanation: Select a slice of rows from a specific column of a DataFrame:
End of explanation
np.random.seed(0)
df_8 = pd.DataFrame(np.random.rand(9).reshape((3, 3)),
                 columns=['a', 'b', 'c'])
df_8
np.random.seed(1)
df_9 = pd.DataFrame(np.random.rand(9).reshape((3, 3)),
                 columns=['b', 'c', 'd'])
df_9
df_8 + df_9
Explanation: Arithmetic and Data Alignment
Adding DataFrame objects results in the union of index pairs for rows and columns if the pairs are not the same, resulting in NaN for indices that do not overlap:
End of explanation
df_10 = df_8.add(df_9, fill_value=0)
df_10
Explanation: Set a fill value instead of NaN for indices that do not overlap:
End of explanation
ser_8 = df_10.iloc[0]
df_11 = df_10 - ser_8
df_11
Explanation: Like NumPy, pandas supports arithmetic operations between DataFrames and Series.
Match the index of the Series on the DataFrame's columns, broadcasting down the rows:
End of explanation
ser_9 = pd.Series(range(3), index=['a', 'd', 'e'])
ser_9
df_11 - ser_9
Explanation: Match the index of the Series on the DataFrame's columns, broadcasting down the rows and union the indices that do not match:
End of explanation
df_11 = np.abs(df_11)
df_11
Explanation: Function Application and Mapping
NumPy ufuncs (element-wise array methods) operate on pandas objects:
End of explanation
df_11.apply(sum)
Explanation: Apply a function on 1D arrays to each column:
End of explanation
df_11.apply(sum, axis=1)
Explanation: Apply a function on 1D arrays to each row:
End of explanation
def func_3(x): 
    return '%.2f' %x
df_11.applymap(func_3)
Explanation: Apply an element-wise Python function to a DataFrame:
End of explanation
df_12 = pd.DataFrame(np.arange(12).reshape((3, 4)),
                  index=['three', 'one', 'two'],
                  columns=['c', 'a', 'b', 'd'])
df_12
Explanation: Sorting
End of explanation
df_12.sort_index()
Explanation: Sort a DataFrame by its index:
End of explanation
df_12.sort_index(axis=1, ascending=False)
Explanation: Sort a DataFrame by columns in descending order:
End of explanation
df_12.sort_values(by=['d', 'c'])
Explanation: Sort a DataFrame's values by column:
End of explanation
df_15 = pd.DataFrame(np.random.randn(10, 3),
                     columns=['a', 'b', 'c'])
df_15['cat1'] = (np.random.rand(10) * 3).round(0)
df_15['cat2'] = (np.random.rand(10)).round(0)
df_15
Explanation: Summarizing and Computing Descriptive Statistics
Unlike NumPy arrays, Pandas descriptive statistics automatically exclude missing data.  NaN values are excluded unless the entire row or column is NA.
End of explanation
df_15.sum()
df_15.sum(axis=1)
df_15.mean(axis=0)
Explanation: Sum and Mean
End of explanation
df_15['a'].describe()
df_15['cat1'].value_counts()
Explanation: Descriptive analysis
End of explanation
pd.pivot_table(df_15, index='cat1', aggfunc=np.mean)
Explanation: Pivot tables
group by cat1 and calculate mean
End of explanation
<END_TASK>