Any way to access methods from individual stages in PySpark PipelineModel?
<p>I've created a <code>PipelineModel</code> for doing LDA in Spark 2.0 (via PySpark API):</p>

<pre><code>def create_lda_pipeline(minTokenLength=1, minDF=1, minTF=1, numTopics=10, seed=42, pattern='[\W]+'):
    """
    Create a pipeline for running an LDA model on a corpus. This function does not need data and will not actually do
    any fitting until invoked by the caller.
    Args:
        minTokenLength:
        minDF: minimum number of documents word is present in corpus
        minTF: minimum number of times word is found in a document
        numTopics:
        seed:
        pattern: regular expression to split words

    Returns:
        pipeline: class pyspark.ml.PipelineModel
    """
    reTokenizer = RegexTokenizer(inputCol="text", outputCol="tokens", pattern=pattern, minTokenLength=minTokenLength)
    cntVec = CountVectorizer(inputCol=reTokenizer.getOutputCol(), outputCol="vectors", minDF=minDF, minTF=minTF)
    lda = LDA(k=numTopics, seed=seed, optimizer="em", featuresCol=cntVec.getOutputCol())
    pipeline = Pipeline(stages=[reTokenizer, cntVec, lda])
    return pipeline
</code></pre>

<p>I want to calculate the perplexity on a dataset using the trained model with the <code>LDAModel.logPerplexity()</code> method, so I tried running the following:</p>

<pre><code>try:
    training = get_20_newsgroups_data(test_or_train='test')
    pipeline = create_lda_pipeline(numTopics=20, minDF=3, minTokenLength=5)
    model = pipeline.fit(training)  # train model on training data
    testing = get_20_newsgroups_data(test_or_train='test')
    perplexity = model.logPerplexity(testing)
    pprint(perplexity)
</code></pre>

<p>This just results in the following <code>AttributeError</code>:</p>

<pre class="lang-none prettyprint-override"><code>'PipelineModel' object has no attribute 'logPerplexity'
</code></pre>

<p>I understand why this error happens, since the <code>logPerplexity</code> method belongs to <code>LDAModel</code>, not <code>PipelineModel</code>, but I am wondering if there is a way to access the method from that stage.</p>