<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Overview
This notebook loads the movie rating data from DSX's local storage then it trains an alternating least square (ALS) model using Spark's Machine Learning library (MLlib).<br>
For more information on Spark ALS, see here
Step1:  Split into training and testing
Next we split the data into training and testing data sets
Step2:  Build the recommendation model using ALS on the training data
I've chosen some values for the ALS parameters.  You should probaly experiment with different values.
Step3:  Extract the product (movie) features
Step4:  Simulate a new user rating a movie
<ASSISTANT_TASK:>
Python Code:
from pyspark.mllib.recommendation import Rating
ratingsRDD = sc.textFile('ratings.dat') \
               .map(lambda l: l.split("::")) \
               .map(lambda p: Rating(
                                  user = int(p[0]), 
                                  product = int(p[1]),
                                  rating = float(p[2]), 
                                  )).cache()
Explanation: Overview
This notebook loads the movie rating data from DSX's local storage then it trains an alternating least square (ALS) model using Spark's Machine Learning library (MLlib).<br>
For more information on Spark ALS, see here:
- http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html#collaborative-filtering
- https://github.com/jadianes/spark-movie-lens
Load the data
End of explanation
(training, test) = ratingsRDD.randomSplit([0.8, 0.2])
numTraining = training.count()
numTest = test.count()
# verify row counts for each dataset
print("Total: {0}, Training: {1}, test: {2}".format(ratingsRDD.count(), numTraining, numTest))
Explanation: Split into training and testing
Next we split the data into training and testing data sets
End of explanation
from pyspark.mllib.recommendation import ALS
rank = 50
numIterations = 20
lambdaParam = 0.1
model = ALS.train(training, rank, numIterations, lambdaParam)
Explanation: Build the recommendation model using ALS on the training data
I've chosen some values for the ALS parameters.  You should probaly experiment with different values.
End of explanation
import numpy as np
pf = model.productFeatures().cache()
pf_keys = pf.sortByKey().keys().collect()
pf_vals = pf.sortByKey().map(lambda x: list(x[1])).collect()             
        
Vt = np.matrix(np.asarray(pf.values().collect()))
Explanation: Extract the product (movie) features
End of explanation
full_u = np.zeros(len(pf_keys))
full_u.itemset(1, 5) # user has rated product_id:1 = 5
recommendations = full_u*Vt*Vt.T
print("predicted rating value", np.sort(recommendations)[:,-10:])
top_ten_recommended_product_ids = np.where(recommendations >= np.sort(recommendations)[:,-10:].min())[1]
print("predict rating prod_id", np.array_repr(top_ten_recommended_product_ids))
Explanation: Simulate a new user rating a movie
End of explanation
<END_TASK>