 Raised in Charlotte, Simmons was the son of a builder and former football coach of Davidson College.Nicknamed "Chunk" by a nurse for his size when still a baby, Simmons played football for Central High School where he graduated in 1942.During World War II he was wounded in action in Italy whilst serving with the 10th Mountain Division.