Take the 2-minute tour ×

I have a floating point grid/raster and will want to make all values rounded to the nearest tenth.

I have converted to integer (added 0.5 for rounding), but the final floating point output grid is not giving me the exact values I want. For e.g. 4676.0000 should become 467.6, but instead I get 467.60006.

Also 4674.00000 should become 467.4 but instead I get 467.399994.

Any help and suggestions?

share|improve this question
add comment

1 Answer 1

This is an inherent limitation of IEEE single-precision floating point format, which uses binary instead of base 10. Your options include:

(i) Don't be concerned, because the errors are so small. (They will be limited to less than one one-millionth of the size of each number.) However, be cautious when making comparisons. For example, if you want to select all cells with value between 467.4, the query

    [MyGrid] == 467.4

will fail to select anything. (There are similar problems with SQL in ArcGIS when querying fields of type "double" and "float".) When working with floats where this error matters, make comparisons with a small tolerance, as in

    Abs([MyGrid] - 467.4) < 0.001

or, better yet,

    Abs([MyGrid] - 467.4) < (0.000001 * 467.4)

(ii) Keep the grid in integer format before dividing by 10. Do the division by 10 only at the very end. This works for grid processing that is linear. (A grid operation f([aGrid]) is linear when f(a * [aGrid]) = a * f([aGrid]) for any number 'a'.) Many grid operations are linear, including most focal statistics (but not the variance), multiplication, addition, etc. Other operations are not linear but have predictable results that can be corrected later. For example, the focal variance of a*[aGrid] equals a^2 times the focal variance of [aGrid]. You have to be most careful when deriving slopes, aspects, and curvatures: those can be irremediably affected unless you simultaneously rescale the horizontal units of measurement.

(iii) Return to the original integer grid, then round all values to a power of 2 rather than a power of 10. Here, you might choose to round all values to the nearest eighth or sixteenth. Their floating point representations will be exact.

It's hard to imagine situations where the imprecision is greater than the inherent inaccuracies in the raster data themselves, so (i) is usually the way to go.

share|improve this answer
add comment

Your Answer


