Precision of double values in Spark
<p>I am reading some data from a CSV file, and I have custom code to parse string values into different data types. For numbers, I use:</p>

<pre><code>val format = NumberFormat.getNumberInstance()
</code></pre>

<p>which returns a <code>DecimalFormat</code>, and I call <code>parse</code> function on that to get my numeric value. <code>DecimalFormat</code> has arbitrary precision, so I am not losing any precision there. However, when the data is pushed into a Spark DataFrame, it is stored using <code>DoubleType</code>. At this point, I am expecting to see some precision issues, however I do not. I tried entering values from 0.1, 0.01, 0.001, ..., 1e-11 in my CSV file, and when I look at the values stored in the Spark DataFrame, they are all accurately represented (i.e. not like 0.099999999). I am surprised by this behavior since I do not expect a double value to store arbitrary precision. Can anyone help me understand the magic here? </p>

<p>Cheers!</p>