 Baker was born and educated in Birmingham, Warwickshire, and at a boarding school in Kent.He was the son of Ethel (1906–1990), a pianist and dress maker, and Harold Baker (1908–1985), an artist, musician and draughtsman.He went to live with his father, stepmother and half-sister in Hastings, Sussex.