 In 1920, his father deserted the family and moved to the USA.In 1927, Brown left Scotland to search for his father, settling in Westfield, New Jersey and finding work on the production floor of a metal box factory, where his riveting skills were handy.His brother, Jock, also earned a cap with Scotland and won the 1939 Scottish FA Cup with Clyde FC, while youngest brother Tom played professionally in England for Ipswich Town.