 In 1936 he married his first wife, the artist, Betty Fitzgerald, who was later known as Duffy Ayers, and the couple had two children.In 1956 he divorced Duffy and in 1958 married Diana Arnold-Forster.Not long after the 1958 Great Bardfield summer exhibition the couple moved to the nearby village of Stisted, Essex.