sparksql drop hive table
<p>I want to drop a hive table through sparksql.</p>

<p>In a cluster with hadoop 2.6, hive 2.0, spark 1.6 and spark 2.0 installed. I tried the following code in pyspark shell and spark-submit job with both version.</p>

<pre><code>sqlContext.sql('drop table test')  //spark 1.6
spark.sql('drop table test')       //spark 2.0
</code></pre>

<p>Both code works fine in pyspark-shell , I can see from the hive cli that the <strong><em>test</em></strong> table no longer exist.</p>

<p>However, if the code was in a python file and later submitted to cluster using spark-submit, the code never took effect. </p>

<p>spark 2.0 even gave error about </p>

<pre><code>pyspark.sql.utils.AnalysisException: u"Table to drop '`try`' does not exist;"
</code></pre>

<p>I have copied <strong>hive-site.xml</strong> into the conf directory in spark.</p>

<p>What would be the correct way to drop a hive table through sparksql?</p>

<p>Update:</p>

<p>I tried compared the spark environment between the spark-shell and the job I submitted using the following code</p>

<pre><code>spark-submit --master yarn --deploy-mode cluster try_spark_sql.py
</code></pre>

<p>In the spark-shell environment, I can see <strong>spark.sql.catalogImplementation</strong>   is set to <strong>hive</strong></p>

<p>IN the job submitted using the above code. The environment doesn't contain <strong>spark.sql.catalogImplementation</strong>
I tried setting it using the following code:</p>

<pre><code>spark = SparkSession.builder.appName("PythonSQL").config("spark.sql.catalogImplementation","hive").
</code></pre>

<p>But it doesn't have any effect on the environment.</p>

<p>One workaround I found is submitting the job using <strong>client</strong> mode instead of <strong>cluster</strong> mode. Then the hive table can be successfully dropped.</p>