How does computing table stats in hive or impala speed up queries in Spark SQL?
<p>For increasing performance (e.g. for joins) it is recommended to compute table statics first.  </p>

<p>In Hive I can do::</p>

<pre><code>analyze table &lt;table name&gt; compute statistics;
</code></pre>

<p>In Impala:</p>

<pre><code>compute stats &lt;table name&gt;;
</code></pre>

<p>Does my spark application (reading from hive-tables) also benefit from pre-computed statistics? If yes, which one do I need to run? Are they both saving the stats in the hive metastore? I'm using spark 1.6.1 on Cloudera 5.5.4</p>

<p>Note:
In the Docs of spark 1.6.1 (<a href="https://spark.apache.org/docs/1.6.1/sql-programming-guide.html" rel="noreferrer">https://spark.apache.org/docs/1.6.1/sql-programming-guide.html</a>) for the parameter <code>spark.sql.autoBroadcastJoinThreshold</code> I found a hint: </p>

<blockquote>
  <p>Note that currently statistics are only supported for Hive Metastore
  tables where the command ANALYZE TABLE  COMPUTE STATISTICS
  noscan has been run.</p>
</blockquote>