 He began playing the drums and piano at an early age.Before studying music in earnest, he was a boxer, and was a successful Golden Gloves champion.During the late '50s, Lytle continued to box, but landed jobs as a drummer for Ray Charles, Jimmy Witherspoon and Gene Ammons.