spark - Call unbound method with explicit self error
<p>How I can pass a python function on Spark map() operation?</p>

<p>I want to reuse the same sc, sqlCtx and df at all calls, but I got <em>Call unbound method with explicit self error</em>.</p>

<pre class="lang-py prettyprint-override"><code># initialize spark
conf = SparkConf().setAppName("spark script")
conf = conf.setMaster("local[*]")
sc = SparkContext(conf=conf, pyFiles=[os.path.realpath(__file__)])
sqlCtx = SQLContext(sc)
sqlCtx.setConf("spark.sql.parquet.binaryAstext","true")

df = sqlCtx.read.load(s3_url_to_parquet_file)
df.registerTempTable("temp_table")

def parse_line(log_line):
    dict_with_line_info = {}
    # ... parse operations

    id = df.filter(df["SessionKey"] == session_token).show()
    dict_with_line_info.update({"user_id":id})
    return dict_with_line_info

def main(log_line):
    s3_url_to_tar_file = "s3n://....."
    log = sc.textFile(s3_url_to_tar_file)
    return log.map(lambda log_line: parse_line(log_line)).collect()

if __name__ == '__main__':
    for log_line in log_text:
        main(x)
</code></pre>