Hive configuration for Spark integration tests
<p>I am looking for a way to configure Hive for Spark SQL integration testing such that tables are written either in a temporary directory or somewhere under the test root. My investigation suggests that this requires setting both <code>fs.defaultFS</code> and <code>hive.metastore.warehouse.dir</code> before <code>HiveContext</code> is created. </p>

<p>Just setting the latter, as mentioned in this <a href="https://stackoverflow.com/a/33248723/622495">answer</a> is not working on Spark 1.6.1. </p>

<pre><code>val sqlc = new HiveContext(sparkContext)
sqlc.setConf("hive.metastore.warehouse.dir", hiveWarehouseDir)
</code></pre>

<p>The table metadata goes in the right place but the written files go to /user/hive/warehouse.</p>

<p>If a dataframe is saved without an explicit path, e.g.,</p>

<pre><code>df.write.saveAsTable("tbl")
</code></pre>

<p>the location to write files to is determined via a call to <code>HiveMetastoreCatalog.hiveDefaultTableFilePath</code>, which uses the <code>location</code> of the default database, which seems to be cached during the <code>HiveContext</code> construction, thus setting <code>fs.defaultFS</code> after <code>HiveContext</code> construction has no effect.</p>

<p>As an aside, but very relevant for integration testing, this also means that <code>DROP TABLE tbl</code> only removes the table metadata but leaves the table files, which wreaks havoc with expectations. This is a known problem--see <a href="http://markmail.org/message/56ub3vgg5sck2j43" rel="nofollow noreferrer">here</a> &amp; <a href="http://markmail.org/message/56ub3vgg5sck2j43" rel="nofollow noreferrer">here</a>--and the solution may be to ensure that <code>hive.metastore.warehouse.dir</code> == <code>fs.defaultFS</code> + <code>user/hive/warehouse</code>.</p>

<p>In short, how can configuration properties such as <code>fs.defaultFS</code> and <code>hive.metastore.warehouse.dir</code> be set programmatically <em>before</em> the <code>HiveContext</code> constructor runs?</p>