Spark lists all leaf node even in partitioned data
<p>I have parquet data partitioned by <code>date</code> &amp; <code>hour</code>, folder structure:</p>

<pre><code>events_v3
  -- event_date=2015-01-01
    -- event_hour=2015-01-1
      -- part10000.parquet.gz
  -- event_date=2015-01-02
    -- event_hour=5
      -- part10000.parquet.gz
</code></pre>

<p>I have created a table <code>raw_events</code> via spark but when I try to query, it scans all the directories for footer and that slows down the initial query, even if I am querying only one day worth of data.</p>

<p>query:
<code>select * from raw_events where event_date='2016-01-01'</code></p>

<p>similar problem : <a href="http://mail-archives.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAAswR-7Qbd2tdLSsO76zyw9tvs-Njw2YVd36bRfCG3DKZrH0tw@mail.gmail.com%3E" rel="noreferrer">http://mail-archives.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAAswR-7Qbd2tdLSsO76zyw9tvs-Njw2YVd36bRfCG3DKZrH0tw@mail.gmail.com%3E</a> ( but its old)</p>

<p>Log:</p>

<pre><code>App &gt; 16/09/15 03:14:03 main INFO HadoopFsRelation: Listing leaf files and directories in parallel under: s3a://bucket/events_v3/
</code></pre>

<p>and then it spawns 350 tasks since there are 350 days worth of data.</p>

<p>I have disabled <code>schemaMerge</code>, and have also specified the schema to read as, so it can just go to the partition that I am looking at, why should it print all the leaf files ?
Listing leaf files with 2 executors take 10 minutes, and the query actual execution takes on 20 seconds</p>

<p>code sample:</p>

<pre><code>val sparkSession = org.apache.spark.sql.SparkSession.builder.getOrCreate()
val df = sparkSession.read.option("mergeSchema","false").format("parquet").load("s3a://bucket/events_v3")
    df.createOrReplaceTempView("temp_events")
    sparkSession.sql(
      """
        |select verb,count(*) from temp_events where event_date = "2016-01-01" group by verb
      """.stripMargin).show()
</code></pre>