 Abergavenny also served as Recorder of Harwich.Around 1790, he rebuilt the family's old house at Eridge and made it his principal seat.In 1803 or 1805, having moved to Eridge Castle, he sold his estate at Kidbrooke, near East Grinstead, to Charles Abbot.