I'm sure there are several reasons why this is the case, but the reason I can think of is that the Rift wasn't be tracked, so where my head is located relative to where my hands are located isn't matching up.