 He lived at Delgatie Castle near Turriff, Aberdeenshire.He died at Percy Cross, Walham Green, Fulham, Middlesex in March 1851.He had married Anne Stein, by whom he had two sons and two daughters, including James Duff, 5th Earl Fife and George Skene Duff.