 Brown grew up in Margate in Kent and served as general secretary of the Civil Service Clerical Association from 1919 to 1942.He joined the Labour Party and stood for several seats before he was elected at the 1929 general election as a Labour MP for Wolverhampton West.In 1931, he resigned the Labour whip, and joined the New Party led by Oswald Mosley.