<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <table>
<tr>
<td width=15%><img src="./img/UGA.png"></img></td>
<td><center><h1>Introduction to Python for Data Sciences</h1></center></td>
<td width=15%><a href="http
Step1:  Numpy operations
If we apply a NumPy function on a Pandas datframe, the result will be another Pandas dataframe with the indices preserved.
Step2:  Arithmetic operations
Arithmetic operations can also be performed either with <tt>+ - / *</tt> or with dedicated <tt>add multiply</tt> etc methods
Step3:  The pandas arithmetic functions also have an option to fill missing values by replacing the missing one in either of the dataframes by some value.
Step4:  Appending, Concatenating, and Merging
Thanks to naming, dataframes can be easily added, merged, etc. However, if some entries are missing (columns or indices), the operations may get complicated. Here the most standard situations are covered, take a look at the documentation (notably this one on merging, appending, and concatenating )
Appending is for adding the lines of one dataframe with another one with the same columns.
Step5:  Sometimes, indexes do not matter, they can be resetted using <tt>ignore_index=True</tt>.
Step6:  Concatenating is for adding lines and/or columns of multiples datasets (it is a generalization of appending)
Step7:  The most important settings of the <tt>concat</tt> function are <tt>pd.concat(objs, axis=0, join='outer',ignore_index=False)</tt> where <br/>
. objs is the list of dataframes to concatenate <br/>
. axis is the axis on which to concatenate 0 (default) for the lines and 1 for the columns <br/>
. join is to decide if we keep all columns/indices on the other axis ('outer' ,default), or the intersection ( 'inner') <br/>
. ignore_index is to decide is we keep the previous names (False, default) or give new ones (True)
For a detailed view see  this doc on merging, appending, and concatenating
Step8:  Merging is for putting together two dataframes with hopefully common data
For a detailed view see  this doc on merging, appending, and concatenating
Step9:  Preparing the Data
Before exploring the data, it is primordial to verify its soundness, indeed if it has missing or replicated data, the results of our test may not be accurate. Pandas provides a collection of methodes to verify the sanity of the data (recall that when data is missing for an entry, it is noted as NaN, and thus any further operation including this will be NaN). 
To explore some typical problems in a dataset, I messed with a small part of the MovieLens dataset. The ratings_mess.csv file contains 4 columns
Step10:  Missing values
Pandas provides functions that check if the values are missing
Step11:  Carefully pruning data
Now that we have to prune lines of our data, this will be done using dropna() through dataframe.dropna(subset=["col_1","col_2"],inplace=True) which drops all rows with at least one missing value in the columns col1, col2 of dataframe in place that is without copy.
Warning
Step12:  To fill missing data (from a certain column), the recommended way is to use fillna() through dataframe["col"].fillna(value,inplace=True) which replace all missing values in the column col of dataframe by value in place that is without copy (again this is irreversible, to use the copy version use inplace=False).
Step13:  This indeed gives the correct result, however, the line indexing presents missing number. The indexes can be resetted with  reset_index(inplace=True,drop=True)
Step14:  Improper values
Even without the missing values, some lines are problematic as they feature values outside of prescribed range (userId id of the user, integer greater than 1; movieId  id of the user, integer greater than 1; rating rating of the user to the movie, float between 0.0 and 5.0; imestamp timestamp, integer )
Step15:  Now, we drop the corresponding line, with drop by drop(problematic_row.index, inplace=True). 
Warning
Step16:  And finally the ratings.
Step17:  We finally have our dataset cured! Let us save it for further use.
to_csv saves as CSV into some file, index=False drops the index names as we did not specify it.
Step18:  Basic Statistics
With our cured dataset, we can begin exploring.
Step19:  The following table summarizes some other built-in Pandas aggregations
Step20:  We see that these statistics do not make sense for all rows. Let us drop the timestamp and examine the ratings.
Step21:  GroupBy
These ratings are linked to users and movies, in order to have a separate view per user/movie, grouping has to be used.
The GroupBy operation (that comes from SQL) accomplishes
Step22:  So to get the mean of the ratings per user, the command is
Step23:  Filtering
Filtering is the action of deleting rows depending on a boolean function. For instance, the following removes the user with a rating of only one movie.
Step24:  Transformations
Transforming is the actions of applying a transformation (sic).
For instance, let us normalize the ratings so that they have zero mean for each user.
Step25:  Aggregations [*]
Aggregations let you aggreagate several operations.
Step26:  Exercises
Exercise
Step27:  Exercise
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
Explanation: <table>
<tr>
<td width=15%><img src="./img/UGA.png"></img></td>
<td><center><h1>Introduction to Python for Data Sciences</h1></center></td>
<td width=15%><a href="http://www.iutzeler.org" style="font-size: 16px; font-weight: bold">Franck Iutzeler</a> </td>
</tr>
</table>
<br/><br/>
<center><a style="font-size: 40pt; font-weight: bold">Chap. 3 - Data Handling with Pandas </a></center> 
<br/><br/>
2- Dataframes
Operations
End of explanation
df = pd.DataFrame(np.random.randint(0, 10, (3, 4)), columns=['A', 'B', 'C', 'D'])
df
np.cos(df * np.pi/2 ) - 1
Explanation: Numpy operations
If we apply a NumPy function on a Pandas datframe, the result will be another Pandas dataframe with the indices preserved.
End of explanation
A = pd.DataFrame(np.random.randint(0, 20, (2, 2)), columns=list('AB'))
A
B = pd.DataFrame(np.random.randint(0, 10, (3, 3)), columns=list('BAC'))
B
A+B
Explanation: Arithmetic operations
Arithmetic operations can also be performed either with <tt>+ - / *</tt> or with dedicated <tt>add multiply</tt> etc methods
End of explanation
A.add(B, fill_value=0.0)
Explanation: The pandas arithmetic functions also have an option to fill missing values by replacing the missing one in either of the dataframes by some value.
End of explanation
A = pd.DataFrame(np.random.randint(0, 20, (2, 2)), columns=list('AB'))
A2 = pd.DataFrame(np.random.randint(0, 20, (3, 2)), columns=list('AB'))
print("A:\n",A,"\nA2:\n",A2)
A.append(A2) # this does not "append to A" but creates a new dataframe
Explanation: Appending, Concatenating, and Merging
Thanks to naming, dataframes can be easily added, merged, etc. However, if some entries are missing (columns or indices), the operations may get complicated. Here the most standard situations are covered, take a look at the documentation (notably this one on merging, appending, and concatenating )
Appending is for adding the lines of one dataframe with another one with the same columns.
End of explanation
A.append(A2,ignore_index=True)
Explanation: Sometimes, indexes do not matter, they can be resetted using <tt>ignore_index=True</tt>.
End of explanation
A = pd.DataFrame(np.random.randint(0, 20, (2, 2)), columns=list('AB'))
A2 = pd.DataFrame(np.random.randint(0, 20, (3, 2)), columns=list('AB'))
A3 = pd.DataFrame(np.random.randint(0, 20, (1, 3)), columns=list('CAD'))
print("A:\n",A,"\nA2:\n",A2,"\nA3:\n",A3)
Explanation: Concatenating is for adding lines and/or columns of multiples datasets (it is a generalization of appending)
End of explanation
pd.concat([A,A2,A3],ignore_index=True)
pd.concat([A,A2,A3],axis=1)
pd.concat([A,A2,A3],axis=1,ignore_index=True,join='inner')
Explanation: The most important settings of the <tt>concat</tt> function are <tt>pd.concat(objs, axis=0, join='outer',ignore_index=False)</tt> where <br/>
. objs is the list of dataframes to concatenate <br/>
. axis is the axis on which to concatenate 0 (default) for the lines and 1 for the columns <br/>
. join is to decide if we keep all columns/indices on the other axis ('outer' ,default), or the intersection ( 'inner') <br/>
. ignore_index is to decide is we keep the previous names (False, default) or give new ones (True)
For a detailed view see  this doc on merging, appending, and concatenating
End of explanation
df1 = pd.DataFrame({'employee': ['Bob', 'Jake', 'Lisa', 'Sue'],
                    'group': ['Accounting', 'Engineering', 'Engineering', 'HR']})
df1
df2 = pd.DataFrame({'employee': ['Lisa', 'Bob', 'Jake', 'Sue'],
                    'hire_date': [2004, 2008, 2012, 2014]})
df2
df3 = pd.merge(df1,df2)
df3
df4 = pd.DataFrame({'group': ['Accounting', 'Engineering', 'HR'],
                    'supervisor': ['Carly', 'Guido', 'Steve']})
df4
pd.merge(df3,df4)
Explanation: Merging is for putting together two dataframes with hopefully common data
For a detailed view see  this doc on merging, appending, and concatenating
End of explanation
ratings = pd.read_csv('data/ml-small/ratings_mess.csv')
ratings.head(7) # enables to display the top n lines of a dataframe, 5 by default
Explanation: Preparing the Data
Before exploring the data, it is primordial to verify its soundness, indeed if it has missing or replicated data, the results of our test may not be accurate. Pandas provides a collection of methodes to verify the sanity of the data (recall that when data is missing for an entry, it is noted as NaN, and thus any further operation including this will be NaN). 
To explore some typical problems in a dataset, I messed with a small part of the MovieLens dataset. The ratings_mess.csv file contains 4 columns:
* userId id of the user, integer greater than 1 
* movieId  id of the user, integer greater than 1 
* rating rating of the user to the movie, float between 0.0 and 5.0
* timestamp timestamp, integer 
and features (man-made!) errors, some of them minor some of them major.
End of explanation
ratings.isnull().head(5)
Explanation: Missing values
Pandas provides functions that check if the values are missing:
isnull(): Generate a boolean mask indicating missing values
notnull(): Opposite of isnull()
End of explanation
ratings.dropna(subset=["userId","movieId","rating"],inplace=True)
ratings.head(5)
Explanation: Carefully pruning data
Now that we have to prune lines of our data, this will be done using dropna() through dataframe.dropna(subset=["col_1","col_2"],inplace=True) which drops all rows with at least one missing value in the columns col1, col2 of dataframe in place that is without copy.
Warning: this function deletes any line with at least one missing data, which is not always wishable. Also, with inplace=True, it is applied in place, meaning that they modify the dataframe it is applied to, it is thus an irreversible operation; drop inplace=True to create a copy or see the result before apllying it.
For instance here, userId,movieId,rating are essential whereas the timestamp is not (it can be dropped for the prediciton process). Thus, we will delete the lines where one of userId,movieId,rating is missing and fill the timestamp with 0 when it is missing.
End of explanation
ratings["timestamp"].fillna(0,inplace=True)
ratings.head(7)
Explanation: To fill missing data (from a certain column), the recommended way is to use fillna() through dataframe["col"].fillna(value,inplace=True) which replace all missing values in the column col of dataframe by value in place that is without copy (again this is irreversible, to use the copy version use inplace=False).
End of explanation
ratings.reset_index(inplace=True,drop=True)
ratings.head(7)
Explanation: This indeed gives the correct result, however, the line indexing presents missing number. The indexes can be resetted with  reset_index(inplace=True,drop=True)
End of explanation
ratings[ratings["userId"]<1]  # Identifying a problem
Explanation: Improper values
Even without the missing values, some lines are problematic as they feature values outside of prescribed range (userId id of the user, integer greater than 1; movieId  id of the user, integer greater than 1; rating rating of the user to the movie, float between 0.0 and 5.0; imestamp timestamp, integer )
End of explanation
ratings.drop(ratings[ratings["userId"]<1].index, inplace=True)
ratings.head(7)
pb_rows = ratings[ratings["movieId"]<1]
pb_rows
ratings.drop(pb_rows.index, inplace=True)
Explanation: Now, we drop the corresponding line, with drop by drop(problematic_row.index, inplace=True). 
Warning: Do not forget .index and inplace=True
End of explanation
pb_rows = ratings[ratings["rating"]<0]
pb_rows2 = ratings[ratings["rating"]>5]
tot_pb_rows = pb_rows.append(pb_rows2 )
tot_pb_rows
ratings.drop(tot_pb_rows.index, inplace=True)
ratings.reset_index(inplace=True,drop=True)
Explanation: And finally the ratings.
End of explanation
ratings.to_csv("data/ml-small/ratings_cured.csv",index=False)
Explanation: We finally have our dataset cured! Let us save it for further use.
to_csv saves as CSV into some file, index=False drops the index names as we did not specify it.
End of explanation
ratings = pd.read_csv('data/ml-small/ratings_cured.csv')
ratings.head()
Explanation: Basic Statistics
With our cured dataset, we can begin exploring.
End of explanation
ratings.describe()
Explanation: The following table summarizes some other built-in Pandas aggregations:
| Aggregation              | Description                     |
|--------------------------|---------------------------------|
| count()              | Total number of items           |
| first(), last()  | First and last item             |
| mean(), median() | Mean and median                 |
| min(), max()     | Minimum and maximum             |
| std(), var()     | Standard deviation and variance |
| mad()                | Mean absolute deviation         |
| prod()               | Product of all items            |
| sum()                | Sum of all items                |
These are all methods of DataFrame and Series objects, and description also provides a quick overview.
End of explanation
ratings.drop("timestamp",axis=1,inplace=True)
ratings.head()
ratings["rating"].describe()
Explanation: We see that these statistics do not make sense for all rows. Let us drop the timestamp and examine the ratings.
End of explanation
ratings.head()
Explanation: GroupBy
These ratings are linked to users and movies, in order to have a separate view per user/movie, grouping has to be used.
The GroupBy operation (that comes from SQL) accomplishes:
The split step involves breaking up and grouping a DataFrame depending on the value of the specified key.
The apply step involves computing some function, usually an sum, median, means etc within the individual groups.
The combine step merges the results of these operations into an output array.
<img src="img/GroupBy.png">
<p style="text-align: right">Source: [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by Jake VanderPlas</p>
End of explanation
ratings.groupby("userId")["rating"].mean()
Explanation: So to get the mean of the ratings per user, the command is
End of explanation
ratings.groupby("userId")["rating"].count()
def filter_func(x):
    return x["rating"].count() >= 2
filtered = ratings.groupby("userId").filter(filter_func)
filtered
filtered.groupby("userId")["rating"].count()
Explanation: Filtering
Filtering is the action of deleting rows depending on a boolean function. For instance, the following removes the user with a rating of only one movie.
End of explanation
ratings.groupby("userId")["rating"].mean()
def center_ratings(x):
    x["rating"] = x["rating"] - x["rating"].mean()
    return x
centered = ratings.groupby("userId").apply(center_ratings)
centered.groupby("userId")["rating"].mean()
Explanation: Transformations
Transforming is the actions of applying a transformation (sic).
For instance, let us normalize the ratings so that they have zero mean for each user.
End of explanation
ratings.groupby("userId")["rating"].aggregate([min,max,np.mean,np.median,len])
Explanation: Aggregations [*]
Aggregations let you aggreagate several operations.
End of explanation
import pandas as pd
import numpy as np
ratings_bots = pd.read_csv('data/ml-small/ratings_bots.csv')
Explanation: Exercises
Exercise: Bots Discovery
In the dataset ratings_bots.csv, some users may be bots. To help a movie sucess they add ratings (favorable ones often). To get a better recommendation, we try to remove them.
Count the users with a mean rating above 4.7/5 and delete them 
hint: the nunique  function may be helpful to count
Delete multiples reviews of a movie by a single user by replacing them with only the first one. What is the proportion of potential bots among the users?
hint: the groupby function can be applied to several columns, also reset_index(drop=True) removes the grouby indexing. 
hint: remember the loc function, e.g.  df.loc[df['userId'] == 128] returns a dataframe of the rows where the userId is 128; and df.loc[df['userId'] == 128].loc[samerev['movieId'] == 3825] returns a dataframe of the rows where the userId is 128 and the movieID is 3825.
In total , 17 ratings have to be removed. For instance, user 128 has 3 ratings of the movie 3825
This dataset has around 100 000 ratings so hand picking won't do!
End of explanation
import pandas as pd
import numpy as np
planets = pd.read_csv('data/planets.csv')
print(planets.shape)
planets.head()
Explanation: Exercise: Planets discovery
We will use the Planets dataset, available via the Seaborn package. It provides information on how astronomers found new planets around stars, exoplanets.
Display median, mean and quantile informations for these planets orbital periods, masses, and distances.
For each method, display statistic on the years planets were discovered using this technique.
End of explanation
<END_TASK>