 Baker was born in Luray, Virginia, and attended Luray High School.He gained early recognition as a left-handed batter and right-handed thrower in amateur clubs and broke into the minor leagues in 1938.In 1943, Baker earned his first trial in the major leagues with the St. Louis Browns, after hitting .326 for a minor league club at San Antonio, Texas, in 1942.