 His father was the second surviving son of Thomas Fitzmaurice, 1st Earl of Kerry, and his wife Anne Petty, daughter of Sir William Petty.His father changed his surname in 1751 on inheriting the estates of his uncle, Henry Petty, 1st Earl of Shelburne, and was himself created Viscount Fitzmaurice in 1751 and Earl of Shelburne in 1753.His elder brother William was an MP before he inherited the family titles on his father's death in 1761; he later became Prime Minister of Great Britain in 1782 and was created Marquess of Lansdowne in 1784.