cast schema of a data frame in Spark and Scala
<p>I want to cast the schema of a dataframe to change the type of some columns
using Spark and Scala.</p>

<p>Specifically I am trying to use as[U] function whose description reads:
"<em>Returns a new Dataset where each record has been mapped on to the specified type. 
The method used to map columns depend on the type of U</em>"</p>

<p>In principle this is exactly what I want,  but I cannot get it to work.</p>

<p>Here is a simple example taken from 
<a href="https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala" rel="nofollow noreferrer">https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala</a></p>

<pre><code>

    // definition of data
    val data = Seq(("a", 1), ("b", 2)).toDF("a", "b")

</code></pre>

<p>As expected the schema of data is:</p>

<pre>

    root
     |-- a: string (nullable = true)
     |-- b: integer (nullable = false)
    </pre>

<p>I would like to cast the column "b" to Double.  So I try the following:</p>

<pre><code>

    import session.implicits._;

    println(" --------------------------- Casting using (String Double)")

    val data_TupleCast=data.as[(String, Double)]
    data_TupleCast.show()
    data_TupleCast.printSchema()

    println(" --------------------------- Casting using ClassData_Double")

    case class ClassData_Double(a: String, b: Double)

    val data_ClassCast= data.as[ClassData_Double]
    data_ClassCast.show()
    data_ClassCast.printSchema()

</code></pre>

<p>As I understand the definition of as[u],  the new DataFrames should have the following schema</p>

<pre>

    root
     |-- a: string (nullable = true)
     |-- b: double (nullable = false)

</pre>

<p>But the output is </p>

<pre>

     --------------------------- Casting using (String Double)
    +---+---+
    |  a|  b|
    +---+---+
    |  a|  1|
    |  b|  2|
    +---+---+

    root
     |-- a: string (nullable = true)
     |-- b: integer (nullable = false)

     --------------------------- Casting using ClassData_Double
    +---+---+
    |  a|  b|
    +---+---+
    |  a|  1|
    |  b|  2|
    +---+---+

    root
     |-- a: string (nullable = true)
     |-- b: integer (nullable = false)

</pre>

<p>which shows that column "b" has not been cast to double.</p>

<p>Any hints on what I am doing wrong?</p>

<p>BTW:  I am aware of the previous post "How to change column types in Spark SQL's DataFrame?" (see <a href="https://stackoverflow.com/questions/29383107/how-to-change-column-types-in-spark-sqls-dataframe">How to change column types in Spark SQL&#39;s DataFrame?</a>).  I know I can change the type of columns one at a time,  but I am looking for a more general solution that changes the schema of the whole data in one shot (and I am trying to understand Spark in the process).</p>