 Patterson attended Jack Yates High School.He accepted a football scholarship from the University of Kansas, where he began his career as a linebacker.As a senior, he was initially moved to defensive end, before being switched to cornerback.