How to create a custom Estimator in PySpark
<p>I am trying to build a simple custom <code>Estimator</code> in PySpark MLlib. I have <a href="https://stackoverflow.com/questions/32331848/create-a-custom-transformer-in-pyspark-ml/32337101?noredirect=1#comment62052435_32337101">here</a> that it is possible to write a custom Transformer but I am not sure how to do it on an <code>Estimator</code>. I also don't understand what <code>@keyword_only</code> does and why do I need so many setters and getters. Scikit-learn seem to have a proper document for custom models (<a href="http://scikit-learn.org/stable/developers/contributing.html#rolling-your-own-estimator" rel="noreferrer">see here</a>) but PySpark doesn't.</p>
<p>Pseudo code of an example model:</p>
<pre><code>class NormalDeviation():
    def __init__(self, threshold = 3):
    def fit(x, y=None):
       self.model = {'mean': x.mean(), 'std': x.std()]
    def predict(x):
       return ((x-self.model['mean']) &gt; self.threshold * self.model['std'])
    def decision_function(x): # does ml-lib support this?
</code></pre>