Take the 2-minute tour ×

I am encountering a situation where I cannot calculate exact sum of a seris of logorithms in calculating entropy. Suppose we have a series of numbers $p_i$ and we want to calculate $\sum_ilog(p_i)$, we should multiply $p_i$ together then take the log, or just take the log of each $p_i$ and calculate the summation over them?

The $p_i$ satisfies that $\sum_i p_i = 1$ and each $p_i$ is between 0 and 1.

Things are, we multiply them together, seems I will encounter the loss of precision and calculate the multiplication will accumulate the loss of precision. But when doing log apart, then each log seems to have loss of precisions.

share|improve this question
You would actually want to find $\sum_i p_i\log p_i$, know? In standard texts, they always find $\log$ of each $p_i$ and multiply by $p_i$ and sum it up. But I do not know which is better and why. –  Ashok Oct 1 '12 at 12:44

2 Answers 2

Using standard computer floating point you will have errors at every step along the way, but they shouldn't be too bad. Your $p_i$'s probably cannot be represented exactly, so even $p_1+p_2$ has some error associated. In 64 bit floats, this amounts to a relative error of order $2^{-53}$, so even a lot of those rarely amount to much. It was worse in the 32 bit days. The log function will have errors, too, but if you use a decent library function it shouldn't be much worse. The risk when you multiply comes if you overflow the available range, but this is unlikely too.

The real problem usually comes when you subtract two nearly equal quantities. In your case the terms all have the same sign, so this is not a problem. If your terms cover several orders of magnitude, it is better to add the small ones first.

share|improve this answer
Related: stackoverflow.com/questions/15858433/… –  leonbloy Oct 31 '14 at 10:22

You can use a high precision calculator such as Smart Math Calculator so that you won't lose any precision no matter which method you use to add them up.

share|improve this answer

Your Answer


