spark-redshift takes a lot of time to write to redshift
<p>I am working on setting up spark streamer with kinesis and redshift. I read data from kinesis after every 10 sec, process it and write it to redshift using spark-redshift lib. </p>

<p>The problem is it is taking hell lot of time to write only 300 rows. </p>

<p>This is what it shows me in the console</p>

<pre><code>[Stage 56:====================================================&gt; (193 + 1) / 200]
</code></pre>

<p>Looking at my logs df.write.format is doing this.</p>

<p>I have spark setup on a machine with 4 gb ram and 2 core amazon EC2, running with --master local[*] mode.</p>

<p>Here is how I create stream</p>

<pre><code>kinesisStream = KinesisUtils.createStream(ssc, APPLICATION_NAME, STREAM_NAME, ENDPOINT, REGION_NAME, INITIAL_POS, CHECKPOINT_INTERVAL, awsAccessKeyId =AWSACCESSID, awsSecretKey=AWSSECRETKEY, storageLevel=STORAGE_LEVEL)    
CHECKPOINT_INTERVAL = 60
storageLevel = memory

kinesisStream.foreachRDD(writeTotable)
def WriteToTable(df, type):
    if type in REDSHIFT_PAGEVIEW_TBL:
        df = df.groupby([COL_STARTTIME, COL_ENDTIME, COL_CUSTOMERID, COL_PROJECTID, COL_FONTTYPE, COL_DOMAINNAME, COL_USERAGENT]).count()
        df = df.withColumnRenamed('count', COL_PAGEVIEWCOUNT)

        # Write back to a table

        url = ("jdbc:redshift://" + REDSHIFT_HOSTNAME + ":" + REDSHIFT_PORT + "/" +   REDSHIFT_DATABASE + "?user=" + REDSHIFT_USERNAME + "&amp;password="+ REDSHIFT_PASSWORD)

        s3Dir = 's3n://' + AWSACCESSID + ':' + AWSSECRETKEY + '@' + BUCKET + '/' + FOLDER

        print 'Start writing to redshift'
        df.write.format("com.databricks.spark.redshift").option("url", url).option("dbtable", REDSHIFT_PAGEVIEW_TBL).option('tempdir', s3Dir).mode('Append').save()

        print 'Finished writing to redshift'
</code></pre>

<p>please let me know the reason for taking this much time</p>