Spark program takes a really long time to complete execution
<p>Spark is supposed to finish data processing at lightening speed. But I guess I'm not using the right functionalities for my program to make Spark work that way. </p>

<p>This is how my program looks like:</p>

<pre><code>from pyspark import SparkContext
from pyspark import SQLContext
from pyspark.sql.types import *
from pyspark.sql import Row
from pyparsing import re
import time

start_time = time.time()
sc = SparkContext("local","test")
sqlContext = SQLContext(sc)

def func1(lines):
    for line in lines:
        qtype = re.search("qtype=(\S+)",line)
        try:
            url = re.search(" url=(\S+)",line)
            url=url.group(1)
        except:
            url="null"

        time = re.search("(^\S+)",line)
        .... #extracting field names
        row = Row(time = time.group(1),ttms = ttms.group(1),chi = chi.group(1),pssc = pssc.group(1),cqhm = cqhm.group(1),rtype = rtype.group(1),rdetails = rdetails.group(1),rurl = rurl.group(1),qtype = qtype.group(1),phn = phn.group(1),fqdn = fqdn,url = url)

    return row

file = sc.textFile("C:\\Logs\\filename.log").cache()
result = file.map(lambda x: x.split("\n"))
lines = result.map(func1).collect()
df = sqlContext.createDataFrame(lines)
df.registerTempTable("df")
df.show()

for line in df :
    query1 = sqlContext.sql("SELECT qtype,rtype,COUNT(qtype,rtype) from df GROUP BY qtype,rtype").collect()
    query2 = sqlContext.sql("SELECT qtype, COUNT(qtype) from df GROUP BY qtype").collect()
    query3 = sqlContext.sql("SELECT rtype, COUNT(rtype) from df GROUP BY rtype").collect()
    query4 = sqlContext.sql("SELECT rtype, COUNT(rtype) from df WHERE qtype = \"HTTP\" GROUP BY rtype").collect()

df1 = sqlContext.createDataFrame(query1)
df2 = sqlContext.createDataFrame(query2)
df3 = sqlContext.createDataFrame(query3)
df4 = sqlContext.createDataFrame(query4)
df1.toPandas().to_csv("C:\\Sample logs\\sqlcsv1.csv")
df2.toPandas().to_csv("C:\\Sample logs\\sqlcsv2.csv")
df3.toPandas().to_csv("C:\\Sample logs\\sqlcsv3.csv")
df4.toPandas().to_csv("C:\\Sample logs\\sqlcsv4.csv")
print(time.time() - start_time)
</code></pre>

<p>This program takes almost 200 seconds to execute which is a very long time. I can't figure out the reason. (My log file contains around 34k log lines). I tried using spark's <code>filter</code> for regex expressions but I get the error that <code>rdd is not iterable</code>. So I need to know in what way can I optimize my program to make it run faster.
Also, I get the warning that <code>stage x contains task of very large size</code>. Tried broadcasting <code>lines</code> but gave an error.</p>