How to convert column values from string to decimal?
<p>I'm having a dataframe which contains a really big integer value, example:</p>

<pre><code>42306810747081022358
</code></pre>

<p>When I've tried to convert it to long it was working in the Java but not under the spark envrironment, I was getting </p>

<pre><code>   NumberFormatException: For input string("42306810747081022358")
</code></pre>

<p>Then I tried to convert it too Decimal (BigDecimal) value. Again, easily can do it in Java, but in Spark: 
     dframe.withColumn("c_number",col("c_a").cast(new DecimalType()));</p>

<p>This way I don't get any exceptions, however I can see that all result values are null. </p>

<p>I also tried to use UDF for this purpose but get the same results: </p>

<pre><code>UDF1 cTransformer = new UDF1&lt;String, BigDecimal&gt;() {
        @Override
        public BigDecimal call(String aString) throws Exception {
            return new BigDecimal(aString);
        }
    };
sqlContext.udf().register("cTransformer", cTransformer, new DecimalType());
dframe = dframe.withColumn("c_number", callUDF("cTransformer", dframe.col("c_a"))); 
</code></pre>

<p>And here again all I'm getting is a column with all zeroes. </p>

<p>How should I proceed?</p>