Minecraft wiki entry for Anvil_mechanics, associated text: When combining two items with prior work penalties, while the penalties for both items apply to the cost, only the higher of the penalties of the two items is considered when determining the penalty of the resulting item. For example, when combining two items with 2 workings each, the resulting item has only 3 workings with the fourth consumed by the penalty.
The choice of which item to use as the sacrifice matters. For example having a Soul Speed III book in the first slot and a Mending book in the second slot has a cost of 2 levels, but reversing the order of the books results in a cost of 12, even though the resulting book is the same in the two cases.