Hive on Spark list all partitions for specific hive table and adding a partition
<p>I am using spark 2.0 and I was wondering ,Is it possible to list all files for specific hive table? If so, I can incrementally update those files directly using spark <code>sc.textFile("file.orc")
</code>.
How can I add a new partition to hive table? is there any api on the hive metastore that I can use from spark?</p>

<p>Is there any way to get the internal hive function that map dataframe <code>row =&gt; partition_path</code></p>

<p>my main reasoning is incremental updates for a table. right now the only way I have figured out is <code>FULL OUTER JOIN</code> SQL +<code>SaveMode.Overwrite</code>, which is not so efficient because he will overwrite  all the table while my main interest is incremental updates for some specific partitions/adding new partition</p>

<p><strong>EDIT</strong>
from what I have saw on the HDFS, when SaveMode.Overwrite spark will emit the table definition i.e <code>CREATE TABLE my_table .... PARTITION BY (month,..)</code>. spark is putting all files under the <code>$HIVE/my_table</code> and not under <code>$HIVE/my_table/month/...</code> which means he is not partitioning the data. when I wrote <code>df.write.partitionBy(...).mode(Overwrite).saveAsTable("my_table")</code> I have saw on hdfs that it is correct. 
I have used <code>SaveMode.Overwrite</code> because I am updating records and not appending data.</p>

<p>I load data using <code>spark.table("my_table")</code> which means spark lazily load the table which is a problem since I don't want to load all the table just part of if.</p>

<p>for the question:</p>

<p>1.Does spark going to shuffle the data because I have used <code>partitionBy()</code> ,or he compares current partition and if its the same he will not shuffle the data.</p>

<p>2.Does spark smart enough to use partition pruning when mutating part from the data i.e just for specific month/year, and apply that change instead of loading all the data? (FULL OUTER JOIN is basically operation that scan all the table)</p>