where p i is the probability that an information source will produce the i th symbol from an r -symbol alphabet and is independent of previous symbols. (thus i runs from 1 to r ). H is then a measure of the expected amount of information (log p i) missing before the symbol is known or measured, or, alternatively, the expected amount of information supplied when the symbol becomes known. The set of messages of length N symbols from the source will then have an entropy of NH.