Minecraft wiki entry for Arrow, associated text: If an arrow is stuck in a block, and that block is broken or disappears (e.g., leaves upon decay), then the arrow falls straight down and damages entities below, but never deals a critical hit.