The Entropy of a coin toss

A product is a collection of features, and each feature has mutually exclusive options. If a feature has only two options, then the choice is like a coin toss. The information contained in that choice is measured by entropy.

Entropy is a concept from classical thermodynamics that deals with the amount of disorder in a physical system (see It was extended to information theory by Claude Shannon (see Shannon used entropy as a measure of the amount of information in a message. The simplest example is a coin toss. If we toss a fair coin, there is a 50% chance of getting tails, and a 50% chance of getting heads. Shannon defined the outcome of this experiment as having an entropy, or information content, of one bit. If I send a message (say 0 or 1) to tell you the result (tail or head), that message contains one bit of information.

Things start to get interesting when the coin is not fair. Consider a two-headed coin. The tossing experiment always results in heads, and the message will always be 1. According to Shannon, the information content of this message is zero.

If the coin is weighted so that the probability of tails is 25% and the probability of heads is 75%, then Shannon assigns an entropy of 0.811278. There is some information in knowing the outcome of the coin toss, but not as much as for a fair coin, because we already know that it will probably be heads. The graph below shows the entropy as a function of the probability of getting heads. When this probability is zero or one, the entropy is zero. The entropy reaches its maximum of one when the coin is fair (50%).

Where did the 0.811278 come from? How is the entropy actually computed?

We can’t answer this without introducing logarithms to the base two. In English, two to the third power is eight, so three is the logarithm of eight to the base two. We can write “blog” to mean log to the base 2, or binary log. If p denotes the probability of heads, then entropy is computed by the formula:

Entropy = -p*blog(p) - (1-p)*blog(1-p).

Logarithms to the base 2 arise naturally because one coin toss (2 outcomes) has entropy one, two coin tosses (4 outcomes) has entropy two, three coin tosses (8 outcomes) has entropy three, and so forth.

Read the full posting by Dr. Roy Marsten at :

Comment on this article

Leave a Comment

Connect to this blog to be notified of new entries.
You are not logged in.
Sign In to post unmoderated comments.
Join the community to create your free profile today.
Keyword Tags:  Configurable products   entropy   roy marsten   computational mathematics   probability   demand   forecasting   forecast   BOM   product complexity   inventory   SKU rationalization   forecasting error   capital utilization

Receive the latest blog posts:
Share Your Perspective


TRUSTe Privacy Certification