integrating scikit-learn with pyspark
<p>I'm exploring pyspark and the possibilities of integrating scikit-learn with pyspark. I'd like to train a model on each partition using scikit-learn. That means, when my RDD is is defined and gets distributed among different worker nodes, I'd like to use scikit-learn and train a model (let's say a simple k-means) on each partition which exists on each worker node. As scikit-learn algorithms takes a Pandas dataframe, my initial idea was to call <code>toPandas</code> for each partition and then train my model. However, the <code>toPandas</code> function collects the DataFrame into the driver and this is not something that I'm looking for. Is there any other way to achieve such a goal?</p>