Confusion arises because the term ladder operator is typically used to describe an operator that acts to increment or decrement a quantum number describing the state of a system. To change the state of a particle with the creation/annihilation operators of QFT requires the use of both annihilation and creation operators. An annihilation operator is used to remove a particle from the initial state and a creation operator is used to add a particle to the final state.