How to control partition size in Spark SQL
<p>I have a requirement to load data from an Hive table using Spark SQL <code>HiveContext</code> and load into HDFS. By default, the <code>DataFrame</code> from SQL output is having 2 partitions. To get more parallelism i need more partitions out of the SQL. There is no overloaded method in <code>HiveContex</code>t to take number of partitions parameter.</p>

<p>Repartitioning of the RDD causes shuffling and results in more processing time.</p>

<p>></p>

<pre><code>val result = sqlContext.sql("select * from bt_st_ent")
</code></pre>

<p>Has the log output of:</p>

<pre><code>Starting task 0.0 in stage 131.0 (TID 297, aster1.com, partition 0,NODE_LOCAL, 2203 bytes)
Starting task 1.0 in stage 131.0 (TID 298, aster1.com, partition 1,NODE_LOCAL, 2204 bytes)
</code></pre>

<p>I would like to know is there any way to increase the partitions size of the SQL output.</p>