This peculiar lack of symmetry is related to Thomas precession and the fact that two successive Lorentz transformations rotate the coordinate system. This rotation has no effect on the magnitude of a vector, and hence relative speed is symmetrical.