If the Northerners could stop the Andal invasion at the Neck, why couldn't the Mountain Kings of Vale do so in their guarded passes?
Why did Ned Stark go to the Tower of Joy with so few men? Why not bring a small guard (say 20 more men) of loyal and discreet northerners?