How to build a sparkSession in Spark 2.0 using pyspark?
<p>I just got access to spark 2.0; I have been using spark 1.6.1 up until this point. Can someone please help me set up a sparkSession using pyspark (python)? I know that the scala examples available online are similar (<a href="https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/6122906529858466/431554386690884/4814681571895601/latest.html" rel="noreferrer">here</a>), but I was hoping for a direct walkthrough in python language. </p>

<p>My specific case: I am loading in avro files from S3 in a zeppelin spark notebook. Then building df's and running various pyspark &amp; sql queries off of them. All of my old queries use sqlContext. I know this is poor practice, but I started my notebook with </p>

<p><code>sqlContext = SparkSession.builder.enableHiveSupport().getOrCreate()</code>. </p>

<p>I can read in the avros with </p>

<p><code>mydata =  sqlContext.read.format("com.databricks.spark.avro").load("s3:...</code> </p>

<p>and build dataframes with no issues. But once I start querying the dataframes/temp tables, I keep getting the "java.lang.NullPointerException" error. I think that is indicative of a translational error (e.g. old queries worked in 1.6.1 but need to be tweaked for 2.0). The error occurs regardless of query type. So I am assuming </p>

<p>1.) the sqlContext alias is a bad idea </p>

<p>and </p>

<p>2.) I need to properly set up a sparkSession. </p>

<p>So if someone could show me how this is done, or perhaps explain the discrepancies they know of between the different versions of spark, I would greatly appreciate it. Please let me know if I need to elaborate on this question. I apologize if it is convoluted. </p>