 He had a younger brother Richard, who also became a noted statesman and soldier, and two younger sisters, Mary and Louisa.He succeeded to his father's title of earl of Upper Ossory in 1758 but as this was a title in the Irish peerage it did not entitle him to a seat in the British House of Lords.In 1767 he was instead elected to the House of Commons for Bedfordshire, a seat he held until 1794.