Spark Streaming Job time increasing gradually
<p>I have a streaming job ,</p>

<pre><code>1. Load the data from hdfs file and register it as a temp table.
2. Then join the temp table with tables present in the hive database.
3. Then send the record to Kafka.
</code></pre>

<p>initially it was taking 12s to do complete first cycle , then it increases to the 50s after 10 hrs. I don't understand the issue here. Also i noticed that shuffle write is also increasing in every node after 10 hrs it is 200GB+</p>

<p>Sample code is ,</p>

<pre><code>val rowRDD = hContext.read.format("com.databricks.spark.csv").option("header", "false").
          option("delimiter", delimiter).load(path).map(col =&gt; dosomething)
//Add filter to rdd to convert the data in a time range.
val filteredRDD = rowRDD.filter { col =&gt;{dosomething}}
//Create the data from changed data RDD and schema to new DataFrame
val tblDF = hContext.createDataFrame(filteredRDD, tblSchema).where("crud_status IN ('U','D','I')")
//Register all the records into the temporary tables.
tblDF.registerTempTable("name_changed")
val userDF = hContext.sql("SELECT id,name,account FROM name_changed JOIN account ON(name_changed.id=account.id) JOIN question 
        on (account.question=question.question)")
userDF.foreachPartition { records =&gt;
        val producer = getKafkaProducer(kafkaBootstrap)
        records.foreach { rowData =&gt;
            producer.send(new ProducerRecord[String, Array[Byte]](topicName, rowData) )
          }
        }
        producer.close()
}
</code></pre>