XGBoost-4j by DMLC on Spark-1.6.1
<p>I am trying to use the XGBoost implementation by DMLC on Spark-1.6.1. I am able to train my data with XGBoost but facing difficulties in prediction. I actually want to do prediction in the way it can be done in Apache Spark mllib libraries, that helps with calculation of training error,precision, recall, specificity etc.</p>

<p>I am posting the codes below, also the error I am getting.
I used this xgboost4j-spark-0.5-jar-with-dependencies.jar in spark-shell to start.</p>

<pre><code>import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.SparkContext._
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import ml.dmlc.xgboost4j.scala.Booster
import ml.dmlc.xgboost4j.scala.spark.XGBoost
import ml.dmlc.xgboost4j.scala.DMatrix
import ml.dmlc.xgboost4j.scala.{Booster, DMatrix}
import ml.dmlc.xgboost4j.scala.spark.{DataUtils, XGBoost}
import org.apache.spark.{SparkConf, SparkContext}




//Load and parse the data file.
val data = sc.textFile("file:///home/partha/credit_approval_2_attr.csv")
val data1 = sc.textFile("file:///home/partha/credit_app_fea.csv")


val parsedData = data.map { line =&gt;
    val parts = line.split(',').map(_.toDouble)
    LabeledPoint(parts(0), Vectors.dense(parts.tail))
}.cache()

val parsedData1 = data1.map { line =&gt;
    val parts = line.split(',').map(_.toDouble)
    Vectors.dense(parts)
}



//Tuning Parameters
val paramMap = List(
      "eta" -&gt; 0.1f,  
      "max_depth" -&gt; 5,
      "num_class" -&gt; 2,
      "objective" -&gt; "multi:softmax" ,
      "colsample_bytree" -&gt; 0.8,
       "alpha" -&gt; 1,
       "subsample" -&gt; 0.5).toMap

  //Training the model  
val numRound = 20
val model = XGBoost.train(parsedData, paramMap, numRound, nWorkers = 1)
val pred = model.predict(parsedData1)
pred.collect()
</code></pre>

<p>Output from pred:</p>

<pre><code>res0: Array[Array[Array[Float]]] = Array(Array(Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(...
</code></pre>

<p>Now when I am using:</p>

<pre><code>val labelAndPreds = parsedData.map { point =&gt;
          val prediction = model.predict(point.features)
          (point.label, prediction)
        }
</code></pre>

<p>Output:</p>

<pre><code>&lt;console&gt;:66: error: overloaded method value predict with alternatives:
  (testSet: ml.dmlc.xgboost4j.scala.DMatrix)Array[Array[Float]] &lt;and&gt;
  (testSet: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector])org.apache.spark.rdd.RDD[Array[Array[Float]]]
 cannot be applied to (org.apache.spark.mllib.linalg.Vector)
                  val prediction = model.predict(point.features)
                                     ^
</code></pre>

<p>And then tried this, since predict requires an RDD[Vector].</p>

<pre><code>val labelAndPreds1 = parsedData.map { point =&gt;
          val prediction = model.predict(Vectors.dense(point.features))
          (point.label, prediction)
        }
</code></pre>

<p>The outcome was:</p>

<pre><code>&lt;console&gt;:66: error: overloaded method value dense with alternatives:
  (values: Array[Double])org.apache.spark.mllib.linalg.Vector &lt;and&gt;
  (firstValue: Double,otherValues: Double*)org.apache.spark.mllib.linalg.Vector
 cannot be applied to (org.apache.spark.mllib.linalg.Vector)
                  val prediction = model.predict(Vectors.dense(point.features))
                                                         ^
</code></pre>

<p>Clearly its an issue of RDD type which I am trying to sort out, this is easy with GBT on spark (<a href="http://spark.apache.org/docs/latest/mllib-ensembles.html#gradient-boosted-trees-gbts" rel="nofollow">http://spark.apache.org/docs/latest/mllib-ensembles.html#gradient-boosted-trees-gbts</a>).</p>

<p>Am I trying to do this the right way?</p>

<p>Any help or suggestion would be awesome.</p>