 In 1799 he acquired the manor of Worlington, near Bury St Edmunds, Suffolk and lived at Worlington Hall, the 16th century manor house.Grey died suddenly in 1801 at his home and was buried at All Saints Church, Worlington.He had married twice; firstly in 1753 Margaret, the daughter of Sir Henry Grey, 1st Baronet, of Howick, Northumberland and secondly, in 1762, Elizabeth Kennedy of Newcastle upon Tyne, with whom he had 2 sons and 2 daughters.