Minecraft wiki entry for Before_1.8, associated text: First, the following costs are added together:
Start with the target's modified Base Value, where cost-per-level is halved to a minimum of 1 per level.
Add the prior-work penalty for the target (and possibly the book)
If the target is gaining any new enchantments, add a cost of (number of new enchantments)×(final number of enchantments - 1) + 1.
For each enchantment level the target will gain, add the cost-per-level halved to a minimum of 1 per level. (save this number)
For each enchantment both target and book already have at max level, add the enchantment cost-per-level once.
If the item was already renamed, and is being renamed again, add 3. (This is half the basic rename cost, rounded down. If you are somehow enchanting an item without durability, only pay 2.)
The above sum is then halved and rounded down. It can round down to 0.
For each book enchantment that's incompatible with the target, add 1 per enchantment level.
For each enchantment level the target will gain, add (again) the cost-per-level halved to a minimum of 1 per level.
If the target is being renamed, add the basic rename cost, that is 7 for items with durability, 5 for other items.