 His father was James B. Frazier, who served as Governor of Tennessee from 1903 to 1905 and as a United States Senator from Tennessee from 1905 to 1911.His mother was Louise Douglas (Keith) Frazier.He was educated in the public schools and Baylor Preparatory School in Chattanooga, Tennessee.