Take the 2-minute tour ×

I am reading through code for optimization routines (Nelder Mead, SQP...). Languages are C++, Python. I observe that often conversion from double to float is performed, or methods are duplicated with double resp. float arguments. Why is it profitable in optimization routines code, and is it significant? In my own code in C++, should I be careful for types double and float and why?

Kind regards.

share|improve this question
If you write your own code in C++ why haven't you tagged the question appropriately ? Am I missing something subtle ? –  High Performance Mark Mar 14 '12 at 20:23
@HighPerformanceMark I am going through code in python (in particular scipy source) and c++, and writting my own routine combining several methods. I should have tagged with c++ as well. –  octoback Mar 14 '12 at 20:32
In addition to the things mentioned, for division and square root, the difference in performance between float and double is quite big. But those operations should be avoided in performance code anyway. –  harold Mar 14 '12 at 21:48
@harold avoid these operators and replace thm by ... what (when it is for example a user-given form to optimize)? –  octoback Mar 14 '12 at 21:55
@dlib it depends, there isn't really a drop-in replacement.. As an example, instead of comparing the lengths of vectors, compare the square of their length. Or when dividing a vector by a scalar, multiply it by its reciprocal (so the division only happens once - but for short vectors in SIMD it only happens once anyway, so it's about vectors longer than that). In SSE there are fast reciprocal square root and reciprocal instructions, with less precision. Sometimes that's enough precision. –  harold Mar 15 '12 at 11:01

3 Answers 3

up vote 6 down vote accepted

Often the choice between double and float is made more on space demands than speed. Modern processors are capable of operating on double quite fast.

Floats may be faster than doubles when using SIMD instructions (such as SSE) which can operate on multiple values at a time. Also if the operations are faster than the memory pipeline, the smaller memory requirements of float will speed things overall.

share|improve this answer
Accuracy is also an important concern. Changing the precision in data types can compromise the accuracy of results in some cases. –  Geoff Oxberry Mar 18 '12 at 1:40

Other times that I've come across the need to consider the choice between double and float types in terms of optimisation include:

  • Networking. Sending double precision data across a socket connection will obviously require more time than sending half that amount of data.
  • Mobile and embedded processors may only be able to handle high speed single precision calculations efficiently on a coprocessor.

As mentioned in another answer, modern desktop processors can handle double precision Processing quite fast. However, you have to ask yourself if the double precision processing is really required. I work with audio, and the only time that I can think of where I would need to process double precision data would be when using high order filters where numerical errors can accumulate. Most of the time this can be avoided by paying more careful attention to the algorithm design. There are, of course, other scientific or engineering applications where double precision data is required in order to correctly represent a huge dynamic range.

Even so, the question of how much effort to spend on considering the data type to use really depends on your target platform. If the platform can crunch through doubles with negligible overhead and you have memory to spare then there is no need to concern yourself. Profile small sections of test code to find out.

share|improve this answer

In certain optimization algorithms, the choice between double and float is not made more on space demands than speed. For example, with penalty or barrier methods that are used for interior point methods in nonlinear optimization, a float has insufficient precision compared to a double, and using floats in the algorithm will yield garbage. For this reason, penalty and barrier methods were not used in the 1960s, but were rediscovered later on with the advent of the double precision data type. (For more on these methods, consult Nonlinear Programming: Sequential Unconstrained Minimization Techniques (Classics in Applied Mathematics) by Fiacco and McCormick.)

Another consideration is the conditioning of the underlying linear systems solved in many optimization algorithms. If the linear systems you're solving in something like a Newton iteration are sufficiently ill-conditioned, you will not be able to obtain an accurate solution to those systems.

Only if the loss in precision will not jeopardize your numerics should you consider replacing doubles with floats; even if space constraints force you to do so, you should make sure that the accuracy of your numerical results is not compromised. Once sufficient accuracy is assured for the problems you're working on, you can then worry about space and performance optimizations. You can use the CUTEr test set to validate your optimization routines.

share|improve this answer

Your Answer


