How to use a Scala class inside Pyspark
<p>I've been searching for a while if there is any way to use a <code>Scala</code> class in <code>Pyspark</code>, and I haven't found any documentation nor guide about this subject.</p>

<p>Let's say I create a simple class in <code>Scala</code> that uses some libraries of <code>apache-spark</code>, something like:</p>

<pre><code>class SimpleClass(sqlContext: SQLContext, df: DataFrame, column: String) {
  def exe(): DataFrame = {
    import sqlContext.implicits._

    df.select(col(column))
  }
}
</code></pre>

<ul>
<li>Is there any possible way to use this class in <code>Pyspark</code>?</li>
<li>Is it too tough?</li>
<li>Do I have to create a <code>.py</code> file?  </li>
<li>Is there any guide that shows how to do that?</li>
</ul>

<p>By the way I also looked at the <code>spark</code> code and I felt a bit lost, and I was incapable of replicating their functionality for my own purpose.</p>