 Moore was born on May 25, 1925 on a farm in Spartanburg, South Carolina to Walter M. "Dick" Moore Sr.; he was one of ten children, including six brothers and three sisters.After attending Jenkins and Cleveland Junior High, he started going to Spartanburg High School.After receiving his driver's license at the age of 14, he and friends Joe Eubanks and Cotton Owens raced their cars in the streets.