Spark MLlib error on trainimplicit
<p>Need help! I am using Spark MLlib, <code>ALS.trainimpilict</code>. When I am doing grid search, in most cases the code works normally, but at certain parameters, it will stop and shows the error message. Like:</p>

<pre><code>......
Rank 40, reg 1.0, alpha 2.0, the RMSE = 29.7147495287 
Rank 40, reg 1.0, alpha 5.0, the RMSE = 30.1937843479
Traceback (most recent call last):
  File "/home/ubuntu/test/als.py", line 270, in &lt;module&gt;

  File "/home/ubuntu/test/als.py", line 125, in __init__self.models_grid_search()
  File "/home/ubuntu/test/als.py", line 195, in models_grid_search
  model = ALS.trainImplicit(self.trainData, rank, iterations=self.iterations,
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/recommendation.py", line 201, in trainImplicit
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 130, in callMLlibFunc
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 123, in callJavaFunc
  File "/usr/local/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/usr/local/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
  py4j.protocol.Py4JJavaError: An error occurred while calling o1336.trainImplicitALSModel.
  :org.apache.spark.SparkException: Job aborted due to stage failure: Task 17 in stage 22096.0 failed 4 times, 
  most recent failure: Lost task 17.3 in stage 22096.0 (TID 25114, 172.31.11.21): java.lang.AssertionError: 
  assertion failed: lapack.dppsv returned 23.
     at scala.Predef$.assert(Predef.scala:179)
     at org.apache.spark.ml.recommendation.ALS$CholeskySolver.solve(ALS.scala:393)
     at org.apache.spark.ml.recommendation.ALS$$anonfun$org$apache$spark$ml$recommendation$ALS$$computeFactors$1.apply(ALS.scala:1170)
     at org.apache.spark.ml.recommendation.ALS$$anonfun$org$apache$spark$ml$recommendation$ALS$$computeFactors$1.apply(ALS.scala:1131)
    .....
</code></pre>

<p>I knew that there was an early post mentioning a solution: <a href="https://stackoverflow.com/questions/31484460/spark-gives-a-stackoverflowerror-when-training-using-als">Spark gives a StackOverflowError when training using ALS</a>. But when I put <code>sc.setCheckpointDir('checkpoint/')</code> or <code>ALS.checkpointInterval = 2</code> ("which prevents the recursion used by the codebase from creating an overflow" as explained by the post) in the code, the execution immediately stops, even not giving any train implicit outcome. The error message is </p>

<pre><code>Traceback (most recent call last):
  File "/home/ubuntu/test/als_implicit.py", line 275, in &lt;module&gt; engine = ImplicitCF(sc, rank=8, seed=5L, iterations=10,reg_parameter=0.06)
  File "/home/ubuntu/test/als_implicit.py", line 129, in __init__self.models_grid_search()
  File "/home/ubuntu/test/als_implicit.py", line 200, in models_grid_search lambda_=reg, blocks=-1, alpha=alphas, nonnegative=False, seed=self.seed)
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/recommendation.py", line 201, in trainImplicit
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 130, in callMLlibFunc
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/mllib/common.py", line 123, in callJavaFunc
  File "/usr/local/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/usr/local/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value py4j.protocol.Py4JJavaError: 
  An error occurred while calling o116.trainImplicitALSModel.: org.apache.spark.SparkException: Checkpoint RDD ReliableCheckpointRDD[232]       
  at aggregate at ALS.scala:1182(0) has different number of partitions from original RDD itemFactors-10 
  MapPartitionsRDD[230] at mapValues at ALS.scala:1131(18)
      at org.apache.spark.rdd.ReliableRDDCheckpointData.doCheckpoint(ReliableRDDCheckpointData.scala:73)
      at org.apache.spark.rdd.RDDCheckpointData.checkpoint(RDDCheckpointData.scala:74)
      at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply$mcV$sp(RDD.scala:1655)
      at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1652)
      ....
</code></pre>

<p>Where should I put the <code>sc.setCheckpointDir('checkpoint/')</code> and <code>ALS.checkpointInterval = 2</code>? Should I put right after sc, or before train model? or after train model? The following is my code (for the later error message, with <code>sc.setCheckpointDir('checkpoint/</code>):</p>

<pre><code>from __future__ import print_function
import sys
from pyspark.mllib.recommendation import ALS
from pyspark.mllib.recommendation import Rating
from pyspark import SparkContext, SparkConf

class ImplicitCF(object):
   def __init__(self, sc, rank, seed, iterations, reg_parameter):
       text = sc.textFile(sys.argv[1], 1)
       sc.setCheckpointDir('checkpoint/')
       self.sc = sc

       self.rank = rank
       self.seed = seed
       self.iterations = iterations
       self.reg = reg_parameter

       self.models_grid_search(self)

   .......

   def models_grid_search(self):
       for reg in [1.0, 2.0, 5.0]:
          for alphas in [0.1, 0.5, 1.0, 2.0, 5.0]:
            model = ALS.trainImplicit(self.trainData, rank=self.rank, iterations=self.iterations, lambda_=reg, blocks=-1, alpha=alphas, nonnegative=False, seed=self.seed)
            #self.sc.setCheckpointDir('checkpoint/')                                                       
            #ALS.checkpointInterval = 2    

   ....              

if __name__ == "__main__":
   sc = SparkContext(appName="implicit_train_test")
   engine = ImplicitCF(sc, rank=8, seed=5L, iterations=10, reg_parameter=0.06)
   sc.stop()
</code></pre>

<p>Thank you very much for any experts' solutions and input. My spark version is <code>v1.5.2</code>. Meanwhile, as shown above, I am using pyspark.</p>