Because innumerable materials are dielectrics or conductors while comparatively few are ferromagnets, the reflection or refraction of EM waves (including light) is more often due to differences in the electric properties of media than to differences in their magnetic properties. That circumstance tends to draw attention to the electric vectors, so that we tend to think of the direction of polarization as the direction of the electric vectors, and the "plane of polarization" as the plane containing the electric vectors and the direction of propagation.