 He was the son of James Anthony, a businessman, and Mary, a homemaker.He grew up in Gum Spring, Virginia, until his parents separated, and his father lost his business.At the age of ten years old, Williams and his brother moved to Muncie, Indiana where his father had family, and he first became aware of his father's African American heritage.