Minecraft wiki entry for Experience, associated text: For fractional values, first multiply this value by the number of smelted items removed from the furnace, then award the player the whole-number part, and if there is a fractional part remaining, this represents the chance of an additional experience point.
For example, when smelting 1 coal ore and removing the coal, the value is 0.1, so this grants a 10% chance of getting 1 experience point.
Or, when smelting 7 cobblestone and removing all 7 stone, the value is 0.1 * 7 = 0.7, so this grants 70% chance of getting 1 experience point.