Minecraft wiki entry for Smelting, associated text: For example, when smelting 1 coal ore and removing the coal, the value is 0.1, so every ten coal you remove grants you one experience point on average.
When smelting 5 sea pickles and removing all 5 lime dye, the value is 0.2 × 5 = 1, so this grants only 1 point.
The fractional experience stays within the furnace when the final total is not an integer, so the leftover experience is attributed to the next round of smelting.