 From 1957 until 1977 he was married to Esther "Nonni" Gordon, who is from Johannesburg, South Africa.They had three children, including the 5th Viscount, Jonathan Herbert Samuel.Samuel later married Heather Cumming in 1981.