Load hive partitioned table to Spark Dataframe
<p>I am using <code>Spark 1.4.1</code> version. I am trying to load a partitioned Hive table in to a DataFrame where in the Hive table is partitioned by the <code>year_week</code> number, at a scenario I might have 104 partitions.</p>

<p>But I could see the DataFrame is getting loaded with the data into 200 partitions and I understand that it is due to the <code>spark.sql.shuffle.partitions</code> set to 200 by default.</p>

<p>I would like to know if there is any good way I can load my Hive table to Spark Dataframe with 104 partitions with making sure that the Dataframe is partitioned by <code>year_week</code> number during the Dataframe load time itself.</p>

<p>The reason for my expectation is that I will be doing few joins with huge volume tables, where all are partitioned by <code>year_week</code> number. So having the Dataframe partitioned by <code>year_week</code> number and loaded accordingly will save me a lot of time from re-partitioning them with <code>year_week</code> number.</p>

<p>Please let me know if you have any suggestions to me.</p>

<p>Thanks.</p>