<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Demonstration of a simple evaluation pipeline
Polara supports various evaluation regimes and can be tuned flexibly to achieve the setup you need.
<div class="alert alert-block alert-warning"> Note, that particular evaluation settings may not be directly supported by some models and may require a certain model modification.</div>
For example, matrix factorization models are not directly applicable in the warm-start scenario (when test users are not present in the training set) until the folding-in technique is implemented for generating recommendations. Keep this in mind when creating your custom solutions.  
<div class="alert alert-block alert-info">In this example we demonstrate basic evaluation of two well-known factorization models - `PureSVD` and `iALS` - in both standard and warm-start settings.</div>
The models can be easily tested in both scenarios without any modification due to automatic handling of the warm-start case, provided by Polara.
Preparing Movielens-1M data
Step1:  Standard scenario with known users
Step2:  Let's check for demonstration purposes that all test users are present in the training set
Step3:  PureSVD
Step4:  iALS
Step5:  The maximum possible number of correct recomendations is
Step6:  Both models correctly retrieve around a quarter of all items. Let's look on the averaged relevance scores
Step7:  Warm-start scenario
This will split test users from the training data.
Step8:  There's no intersection between test and training users
Step9:  <div class="alert alert-block alert-info">Polara makes a certain level of efforts to preserve data sanity and consistency.</div>
For example, as can be seen from the log message above, it filters out the items that are happen to be in the test split but are not a part of the training set.
PureSVD
Step10:  Note, that you do not have to recreate the models as they operate on top of the data_model instance.
In fact, the state change in data_model is synchronized with the dependent models' states. It will force models to rebuild themselves even if you do not explicitly specify it (even though it is recommended to be explicit to conform with the Zen of Python).
iALS
Step11:  The maximum possible number of correct recomendations is
Step12:  Check relevance scores
<ASSISTANT_TASK:>
Python Code:
from polara.recommender.data import RecommenderData
from polara.datasets.movielens import get_movielens_data
data = get_movielens_data() # will automatically download it, or you can specify a path to the local copy
data.head()
data.shape
data_model = RecommenderData(data, 'userid', 'movieid', 'rating', seed=0)
data_model.get_configuration()
Explanation: Demonstration of a simple evaluation pipeline
Polara supports various evaluation regimes and can be tuned flexibly to achieve the setup you need.
<div class="alert alert-block alert-warning"> Note, that particular evaluation settings may not be directly supported by some models and may require a certain model modification.</div>
For example, matrix factorization models are not directly applicable in the warm-start scenario (when test users are not present in the training set) until the folding-in technique is implemented for generating recommendations. Keep this in mind when creating your custom solutions.  
<div class="alert alert-block alert-info">In this example we demonstrate basic evaluation of two well-known factorization models - `PureSVD` and `iALS` - in both standard and warm-start settings.</div>
The models can be easily tested in both scenarios without any modification due to automatic handling of the warm-start case, provided by Polara.
Preparing Movielens-1M data
End of explanation
data_model.random_holdout = True # allow not only top-rated items in evaluation, this reduces evaluation biases
data_model.warm_start = False # standard case
data_model.prepare()
Explanation: Standard scenario with known users
End of explanation
data_model.test.holdout['userid'].isin(data_model.index.userid.training.new).all()
Explanation: Let's check for demonstration purposes that all test users are present in the training set:
End of explanation
from polara.recommender.models import SVDModel
svd = SVDModel(data_model) # create model
svd.switch_positive = 4 # mark ratings below 4 as negative feedback and treat them accordingly in evaluation
svd.build() # fit model
svd.evaluate() # by default it calculates the total number of hits
Explanation: PureSVD
End of explanation
# implicit library must be installed separately, follow instructions at https://github.com/benfred/implicit 
from polara.recommender.external.implicit.ialswrapper import ImplicitALS
als = ImplicitALS(data_model) # create model
als.switch_positive = 4 # same as for PureSVD, affects only evaluation
als.build()
als.evaluate()
Explanation: iALS
End of explanation
data_model.test.holdout.query('rating>=4').shape[0]
Explanation: The maximum possible number of correct recomendations is:
End of explanation
svd.evaluate('relevance')
als.evaluate('relevance')
Explanation: Both models correctly retrieve around a quarter of all items. Let's look on the averaged relevance scores:
End of explanation
data_model.warm_start = True # warm-start case
data_model.prepare()
Explanation: Warm-start scenario
This will split test users from the training data.
End of explanation
data_model.index.userid.test.old.isin(data_model.index.userid.training.old).any()
Explanation: There's no intersection between test and training users:
End of explanation
svd.build()
svd.evaluate()
Explanation: <div class="alert alert-block alert-info">Polara makes a certain level of efforts to preserve data sanity and consistency.</div>
For example, as can be seen from the log message above, it filters out the items that are happen to be in the test split but are not a part of the training set.
PureSVD
End of explanation
als.evaluate()
Explanation: Note, that you do not have to recreate the models as they operate on top of the data_model instance.
In fact, the state change in data_model is synchronized with the dependent models' states. It will force models to rebuild themselves even if you do not explicitly specify it (even though it is recommended to be explicit to conform with the Zen of Python).
iALS
End of explanation
data_model.test.holdout.query('rating>=4').shape[0]
Explanation: The maximum possible number of correct recomendations is:
End of explanation
svd.evaluate('relevance')
als.evaluate('relevance')
Explanation: Check relevance scores:
End of explanation
<END_TASK>