 Born in Leith, Brown was orphaned at the age of 7, along with his five brothers.He entered the coal-mining industry while still at school, but after becoming a Scottish schoolboy international, he decided to pursue a career in football.After playing for Belhavenock, Maryhill and Wishaw Juniors in the Scottish Junior League, he joined Scottish Second Division club East Fife for the 1926–27 season.