 Stephen and his younger brother, the great Whig statesman Charles James Fox (1749–1806), were a great trial to their parents because of their gambling and other habits.He was educated at Eton College.When his father died on 1 July 1774, Stephen inherited his title (Baron Holland, of Foxley (1763)) and then his mother's (Baron Holland, of Holland (1762)) upon her death three weeks later.