Reading parquet files from multiple directories in Pyspark
<p>I need to read parquet files from multiple paths that are not parent or child directories. </p>

<p>for example, </p>

<pre><code>dir1 ---
       |
       ------- dir1_1
       |
       ------- dir1_2
dir2 ---
       |
       ------- dir2_1
       |
       ------- dir2_2
</code></pre>

<p><code>sqlContext.read.parquet(dir1)</code> reads parquet files from dir1_1 and dir1_2</p>

<p>Right now I'm reading each dir and merging dataframes using "unionAll". 
Is there a way to read parquet files from dir1_2 and dir2_1 without using <code>unionAll</code> or is there any fancy way using <code>unionAll</code></p>

<p>Thanks</p>