How to select a subset of fields from an array column in Spark?
<p>Let say I have a DataFrame as follow : </p>

<pre><code>case class SubClass(id:String, size:Int,useless:String)
case class MotherClass(subClasss: Array[SubClass])
val df = sqlContext.createDataFrame(List(
      MotherClass(Array(
        SubClass("1",1,"thisIsUseless"),
        SubClass("2",2,"thisIsUseless"),
        SubClass("3",3,"thisIsUseless")
      )),
      MotherClass(Array(
        SubClass("4",4,"thisIsUseless"),
        SubClass("5",5,"thisIsUseless")
      ))
    ))
</code></pre>

<p>The schema is : </p>

<pre class="lang-none prettyprint-override"><code>root
 |-- subClasss: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- id: string (nullable = true)
 |    |    |-- size: integer (nullable = false)
 |    |    |-- useless: string (nullable = true)
</code></pre>

<p>I'm looking for a way to select only a subset of fields : <code>id</code> and <code>size</code> of the array column <code>subClasss</code>, but with keeping the nested array structure.
The resulting schema would be : </p>

<pre class="lang-none prettyprint-override"><code>root
     |-- subClasss: array (nullable = true)
     |    |-- element: struct (containsNull = true)
     |    |    |-- id: string (nullable = true)
     |    |    |-- size: integer (nullable = false)
</code></pre>

<p>I've tried to do a </p>

<pre><code>df.select("subClasss.id","subClasss.size")
</code></pre>

<p>But this splits the array <code>subClasss</code> in two arrays : </p>

<pre class="lang-none prettyprint-override"><code>root
 |-- id: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- size: array (nullable = true)
 |    |-- element: integer (containsNull = true)
</code></pre>

<p>Is there a way to keep the origin structure and just to eliminate the <code>useless</code> field ? Something that would look like : </p>

<pre><code>df.select("subClasss.[id,size]")
</code></pre>

<p>Thanks for your time.</p>