<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Introduction to Non-Personalized Recommenders
The recommendation problem
Recommenders have been around since at least 1992. Today we see different flavours of recommenders, deployed across different verticals
Step1:  The CourseTalk dataset
Step2:  Using pd.merge we get it all into  one big DataFrame.
Step3:  Collaborative filtering
Step4:  Now let's filter down to courses that received at least 20 ratings (a completely arbitrary number);
To do this, I group the data by course_id and use size() to get a Series of group sizes for each title
Step5:  The index of titles receiving at least 20 ratings can then be used to select rows from mean_ratings above
Step6:  By computing the mean rating for each course, we will order with the highest rating listed first.
Step7:  To see the top courses among Coursera students, we can sort by the 'Coursera' column in descending order
Step8:  Now, let's go further!  How about rank the courses with the highest percentage of ratings that are 4 or higher ?  % of ratings 4+
Let's start with a simple pivoting example that does not involve any aggregation. We can extract a ratings matrix as follows
Step9:  Let's extract only the rating that are 4 or higher.
Step10:  Now picking the number of total ratings for each course and the count of ratings 4+ , we can merge them into one DataFrame.
Step11:  Let's now go easy. Let's count the number of ratings for each course, and order with the most number of ratings.
Step12:  Considering this information we can sort by the most rated ones with highest percentage of 4+ ratings.
Step13:  Finally using the formula above that we learned, let's find out what the courses that most often occur wit the popular MOOC An introduction to Interactive Programming with Python by using the method "x + y/ x" .  For each course, calculate the percentage of Programming with python raters who also rated that course. Order with the highest percentage first, and voilá we have the top 5 moocs.
Step14:  First, let's get only the users that rated the course An Introduction to Interactive Programming in Python
Step15:  Now, for all other courses let's filter out only the ratings from users that  rated the Python course.
Step16:  By applying the division
Step17:  Ordering by the score, highest first excepts the first one which contains the course itself.
<ASSISTANT_TASK:>
Python Code:
from IPython.core.display import Image 
Image(filename='./imgs/recsys_arch.png')
Explanation: Introduction to Non-Personalized Recommenders
The recommendation problem
Recommenders have been around since at least 1992. Today we see different flavours of recommenders, deployed across different verticals: 
Amazon
Netflix
Facebook
Last.fm.
What exactly do they do?
Definitions from the literature
In a typical recommender system people provide recommendations as inputs, which
the system then aggregates and directs to appropriate recipients. -- Resnick
and Varian, 1997
Collaborative filtering simply means that people collaborate to help one
another perform filtering by recording their reactions to documents they read.
-- Goldberg et al, 1992
In its most common formulation, the recommendation problem is reduced to the
problem of estimating ratings for the items that have not been seen by a
user. Intuitively, this estimation is usually based on the ratings given by this
user to other items and on some other information [...] Once we can estimate
ratings for the yet unrated items, we can recommend to the user the item(s) with
the highest estimated rating(s). -- Adomavicius and Tuzhilin, 2005
Driven by computer algorithms, recommenders help consumers
by selecting products they will probably like and might buy
based on their browsing, searches, purchases, and preferences. -- Konstan and Riedl, 2012
Notation
$U$ is the set of users in our domain. Its size is $|U|$.
$I$ is the set of items in our domain. Its size is $|I|$.
$I(u)$ is the set of items that user $u$ has rated.
$-I(u)$ is the complement of $I(u)$ i.e., the set of items not yet seen by user $u$.
$U(i)$ is the set of users that have rated item $i$.
$-U(i)$ is the complement of $U(i)$.
Goal of a recommendation system
$$ 
\newcommand{\argmax}{\mathop{\rm argmax}\nolimits}
\forall{u \in U},\; i^* = \argmax_{i \in -I(u)} [S(u,i)] 
$$
Problem statement
The recommendation problem in its most basic form is quite simple to define:
|-------------------+-----+-----+-----+-----+-----|
| user_id, movie_id | m_1 | m_2 | m_3 | m_4 | m_5 |
|-------------------+-----+-----+-----+-----+-----|
| u_1               | ?   | ?   | 4   | ?   | 1   |
|-------------------+-----+-----+-----+-----+-----|
| u_2               | 3   | ?   | ?   | 2   | 2   |
|-------------------+-----+-----+-----+-----+-----|
| u_3               | 3   | ?   | ?   | ?   | ?   |
|-------------------+-----+-----+-----+-----+-----|
| u_4               | ?   | 1   | 2   | 1   | 1   |
|-------------------+-----+-----+-----+-----+-----|
| u_5               | ?   | ?   | ?   | ?   | ?   |
|-------------------+-----+-----+-----+-----+-----|
| u_6               | 2   | ?   | 2   | ?   | ?   |
|-------------------+-----+-----+-----+-----+-----|
| u_7               | ?   | ?   | ?   | ?   | ?   |
|-------------------+-----+-----+-----+-----+-----|
| u_8               | 3   | 1   | 5   | ?   | ?   |
|-------------------+-----+-----+-----+-----+-----|
| u_9               | ?   | ?   | ?   | ?   | 2   |
|-------------------+-----+-----+-----+-----+-----|
Given a partially filled matrix of ratings ($|U|x|I|$), estimate the missing values.
Challenges
Availability of item metadata
Content-based techniques are limited by the amount of metadata that is available
to describe an item. There are domains in which feature extraction methods are
expensive or time consuming, e.g., processing multimedia data such as graphics,
audio/video streams. In the context of grocery items for example, it's often the
case that item information is only partial or completely missing. Examples
include:
Ingredients
Nutrition facts
Brand
Description
County of origin
New user problem
A user has to have rated a sufficient number of items before a recommender
system can have a good idea of what their preferences are. In a content-based
system, the aggregation function needs ratings to aggregate.
New item problem
Collaborative filters rely on an item being rated by many users to compute
aggregates of those ratings. Think of this as the exact counterpart of the new
user problem for content-based systems.
Data sparsity
When looking at the more general versions of content-based and collaborative
systems, the success of the recommender system depends on the availability of a
critical mass of user/item iteractions. We get a first glance at the data
sparsity problem by quantifying the ratio of existing ratings vs $|U|x|I|$. A
highly sparse matrix of interactions makes it difficult to compute similarities
between users and items. As an example, for a user whose tastes are unusual
compared to the rest of the population, there will not be any other users who
are particularly similar, leading to poor recommendations.
Flow chart: the big picture
End of explanation
import pandas as pd
unames = ['user_id', 'username']
users = pd.read_table('./data/users_set.dat',
                      sep='|', header=None, names=unames)
rnames = ['user_id', 'course_id', 'rating']
ratings = pd.read_table('./data/ratings.dat',
                        sep='|', header=None, names=rnames)
mnames = ['course_id', 'title', 'avg_rating', 'workload', 'university', 'difficulty', 'provider']
courses = pd.read_table('./data/cursos.dat',
                       sep='|', header=None, names=mnames)
# show how one of them looks
ratings.head(10)
# show how one of them looks
users[:5]
courses[:5]
Explanation: The CourseTalk dataset: loading and first look
Loading of the CourseTalk database.
The CourseTalk data is spread across three files. Using the pd.read_table
method we load each file:
End of explanation
coursetalk = pd.merge(pd.merge(ratings, courses), users)
coursetalk
coursetalk.ix[0]
Explanation: Using pd.merge we get it all into  one big DataFrame.
End of explanation
mean_ratings = coursetalk.pivot_table('rating', rows='provider', aggfunc='mean')
mean_ratings.order(ascending=False)
Explanation: Collaborative filtering: generalizations of the aggregation function
Non-personalized recommendations
Groupby
The idea of groupby is that of split-apply-combine:
split data in an object according to a given key;
apply a function to each subset;
combine results into a new object.
To get mean course ratings grouped by the provider, we can use the pivot_table method:
End of explanation
ratings_by_title = coursetalk.groupby('title').size()
ratings_by_title[:10]
active_titles = ratings_by_title.index[ratings_by_title >= 20]
active_titles[:10]
Explanation: Now let's filter down to courses that received at least 20 ratings (a completely arbitrary number);
To do this, I group the data by course_id and use size() to get a Series of group sizes for each title:
End of explanation
mean_ratings = coursetalk.pivot_table('rating', rows='title', aggfunc='mean')
mean_ratings
Explanation: The index of titles receiving at least 20 ratings can then be used to select rows from mean_ratings above:
End of explanation
mean_ratings.ix[active_titles].order(ascending=False)
Explanation: By computing the mean rating for each course, we will order with the highest rating listed first.
End of explanation
mean_ratings = coursetalk.pivot_table('rating', rows='title',cols='provider', aggfunc='mean')
mean_ratings[:10]
mean_ratings['coursera'][active_titles].order(ascending=False)[:10]
Explanation: To see the top courses among Coursera students, we can sort by the 'Coursera' column in descending order:
End of explanation
# transform the ratings frame into a ratings matrix
ratings_mtx_df = coursetalk.pivot_table(values='rating',
                                             rows='user_id',
                                             cols='title')
ratings_mtx_df.ix[ratings_mtx_df.index[:15], ratings_mtx_df.columns[:15]]
Explanation: Now, let's go further!  How about rank the courses with the highest percentage of ratings that are 4 or higher ?  % of ratings 4+
Let's start with a simple pivoting example that does not involve any aggregation. We can extract a ratings matrix as follows:
End of explanation
ratings_gte_4 = ratings_mtx_df[ratings_mtx_df>=4.0]
# with an integer axis index only label-based indexing is possible
ratings_gte_4.ix[ratings_gte_4.index[:15], ratings_gte_4.columns[:15]]
Explanation: Let's extract only the rating that are 4 or higher.
End of explanation
ratings_gte_4_pd = pd.DataFrame({'total': ratings_mtx_df.count(), 'gte_4': ratings_gte_4.count()})
ratings_gte_4_pd.head(10)
ratings_gte_4_pd['gte_4_ratio'] = (ratings_gte_4_pd['gte_4'] * 1.0)/ ratings_gte_4_pd.total
ratings_gte_4_pd.head(10)
ranking = [(title,total,gte_4, score) for title, total, gte_4, score in ratings_gte_4_pd.itertuples()]
for title, total, gte_4, score in sorted(ranking, key=lambda x: (x[3], x[2], x[1])  , reverse=True)[:10]:
    print title, total, gte_4, score
Explanation: Now picking the number of total ratings for each course and the count of ratings 4+ , we can merge them into one DataFrame.
End of explanation
ratings_by_title = coursetalk.groupby('title').size()
ratings_by_title.order(ascending=False)[:10]
Explanation: Let's now go easy. Let's count the number of ratings for each course, and order with the most number of ratings.
End of explanation
for title, total, gte_4, score in sorted(ranking, key=lambda x: (x[2], x[3], x[1])  , reverse=True)[:10]:
    print title, total, gte_4, score
Explanation: Considering this information we can sort by the most rated ones with highest percentage of 4+ ratings.
End of explanation
course_users = coursetalk.pivot_table('rating', rows='title', cols='user_id')
course_users.ix[course_users.index[:15], course_users.columns[:15]]
Explanation: Finally using the formula above that we learned, let's find out what the courses that most often occur wit the popular MOOC An introduction to Interactive Programming with Python by using the method "x + y/ x" .  For each course, calculate the percentage of Programming with python raters who also rated that course. Order with the highest percentage first, and voilá we have the top 5 moocs.
End of explanation
ratings_by_course = coursetalk[coursetalk.title == 'An Introduction to Interactive Programming in Python']
ratings_by_course.set_index('user_id', inplace=True)
Explanation: First, let's get only the users that rated the course An Introduction to Interactive Programming in Python
End of explanation
their_ids = ratings_by_course.index
their_ratings = course_users[their_ids]
course_users[their_ids].ix[course_users[their_ids].index[:15], course_users[their_ids].columns[:15]]
Explanation: Now, for all other courses let's filter out only the ratings from users that  rated the Python course.
End of explanation
course_count =  their_ratings.ix['An Introduction to Interactive Programming in Python'].count()
sims = their_ratings.apply(lambda profile: profile.count() / float(course_count) , axis=1)
Explanation: By applying the division: number of ratings who rated Python Course and the given course / total of ratings who rated the Python Course we have  our percentage.
End of explanation
sims.order(ascending=False)[1:][:10]
Explanation: Ordering by the score, highest first excepts the first one which contains the course itself.
End of explanation
<END_TASK>