Minecraft wiki entry for Defeating_a_woodland_mansion, associated text: Time your hits well against vexes, and retreat if necessary to avoid getting struck.
Use your shield to block any attacks since vexes are hard to hit with their small size and they retreat for a few seconds after attacking.
Against a swarm of vexes, just flee from the scene and ignore them; they drop nothing when killed anyway and will despawn within 2 minutes.