VectorAssembler in spark is very slow, even in trivial cases
<p>I've been using Spark for some data analysis and machine learning.</p>

<p>Having read in some data as trainDF, I construct two pipelines which are logically equivalent, but one of which has a VectorAssembler at the end (which only has one inputCols) to demonstrate the slow down:</p>

<pre><code>scala&gt; val assembler = new VectorAssembler().setInputCols(Array("all_description_features")).setOutputCol("features")
assembler: org.apache.spark.ml.feature.VectorAssembler = vecAssembler_a76e6412bc96

scala&gt; val idfDescription = new IDF().setInputCol("all_description_hashed").setOutputCol("all_description_features")
idfDescription: org.apache.spark.ml.feature.IDF = idf_4b504cf08d86

scala&gt; val descriptionArray = Array(tokensDescription, removerDescription, hashingTFDescription, idfDescription, assembler, lr)
descriptionArray: Array[org.apache.spark.ml.PipelineStage with org.apache.spark.ml.util.DefaultParamsWritable{def copy(extra: org.apache.spark.ml.param.ParamMap): org.apache.spark.ml.PipelineStage with org.apache.spark.ml.util.DefaultParamsWritable{def copy(extra: org.apache.spark.ml.param.ParamMap): org.apache.spark.ml.PipelineStage with org.apache.spark.ml.util.DefaultParamsWritable{def copy(extra: org.apache.spark.ml.param.ParamMap): org.apache.spark.ml.PipelineStage with org.apache.spark.ml.util.DefaultParamsWritable}}}] = Array(regexTok_316674b9209b, stopWords_8ecdf6f09955, hashingTF_48cf3f9cc065, idf_4b504cf08d86, vecAssembler_a76e6412bc96, logreg_f0763c33b304)

scala&gt; val pipeline = new Pipeline().setStages(descriptionArray)
pipeline: org.apache.spark.ml.Pipeline = pipeline_4e462d0ee649

scala&gt; time {pipeline.fit(trainDF)}
16/09/28 13:04:17 WARN Executor: 1 block locks were not released by TID = 9526:
[rdd_38_0]
Elapsed time: 62370646425ns
res94: org.apache.spark.ml.PipelineModel = pipeline_4e462d0ee649

scala&gt; val idfDescription = new IDF().setInputCol("all_description_hashed").setOutputCol("features")
idfDescription: org.apache.spark.ml.feature.IDF = idf_264569f76b23

scala&gt; val descriptionArray = Array(tokensDescription, removerDescription, hashingTFDescription, idfDescription, lr)
descriptionArray: Array[org.apache.spark.ml.PipelineStage with org.apache.spark.ml.util.DefaultParamsWritable{def copy(extra: org.apache.spark.ml.param.ParamMap): org.apache.spark.ml.PipelineStage with org.apache.spark.ml.util.DefaultParamsWritable{def copy(extra: org.apache.spark.ml.param.ParamMap): org.apache.spark.ml.PipelineStage with org.apache.spark.ml.util.DefaultParamsWritable{def copy(extra: org.apache.spark.ml.param.ParamMap): org.apache.spark.ml.PipelineStage with org.apache.spark.ml.util.DefaultParamsWritable}}}] = Array(regexTok_316674b9209b, stopWords_8ecdf6f09955, hashingTF_48cf3f9cc065, idf_264569f76b23, logreg_f0763c33b304)

scala&gt; val pipeline = new Pipeline().setStages(descriptionArray)
pipeline: org.apache.spark.ml.Pipeline = pipeline_758ec8aa3228

scala&gt; time {pipeline.fit(trainDF)}
Elapsed time: 11092968167ns
res95: org.apache.spark.ml.PipelineModel = pipeline_758ec8aa3228
</code></pre>

<p>As you can see the pipeline.fit with the additional VectorAssembler is significantly slower. This is a toy example, but the actual example I'm using would benefit from a VectorAssembler (whereas in this case there is no point in using one) and suffers from a similiar performance impact.</p>

<p>Just wondering if this is to be expected, or whether I am using this wrong. I also notice that with the VectorAssembler I get the warning message about locks not being released which may be related?</p>

<p>Thanks for any assistance and guidance!</p>

<p><strong>Update #1</strong></p>

<p>Some further analysis is showing that the additional time taken is in the logisticRegression fit step, not the actual assembling of features. It is puzzling why this would take longer though as the data it is acting on in both cases is identical (I've proved this to myself by joining the two datasets before they are passed into the fit function, and checking the two feature columns match for all ids).</p>

<p><strong>Update #2</strong></p>

<p>One other thing I noticed is that if I write the two datasets out to disk as parquet (one which has gone through the VectorAssembler, and one which hasn't) the one which went through the VectorAssembler is 10x the size even though they have seemingly identical schema, row count and data.</p>

<p><strong>Update #3</strong></p>

<p>OK - so I think I can see what is going on. Although the data with / without the VectorAssembler is identical, the act of calling transform on the VectorAssembler on my data decorates it with a large amount of (in my case somewhat useless) metadata. This causes the disk size bloat and also presumably the much slower regression due to having to process this additional data.</p>