Minecraft wiki entry for Cactus_farming, associated text: It is also important to choose a good block to break the cactus growth with.
Using a full block such as dirt is inefficient, since the cactus items are likely to land on the cactus (and be destroyed).
The best blocks are those with a thin hitbox such as a fence post, iron bar or glass pane. When a cactus hits a fence post, it is much more likely to fall down to the ground rather than onto the cactus plant.
Blocks without a hitbox, such as fence gates or even signs, will also work, but the build will be more expensive and/or complex.