 Brown married his wife, Joyce, in 1962 and they have three sons (Stuart, Steve and Scott) and a daughter, Susan.His son, Steve, succeeded him as chairman of St Johnstone.Brown is also a racehorse owner and breeder.