 On 1 November 1816, he was appointed rector of the family living of Birling, Kent, and on 23 September 1818, to the vicarage of Frant, Sussex, which his elder brother John had vacated for him.He resigned his livings in 1844 and succeeded his unmarried elder brother, John, as Earl of Abergavenny in 1845.He married Caroline Leeke (d. 9 May 1873) on 7 September 1824, daughter of Ralph Leeke of Longford Hall, Shropshire, and they had the following children