 John Seymour, 4th Duke of Somerset (1629–1675), uncle, inherited the estate in 1671 on the death of his nephew the 3rd Duke, and in 1672 he rebuilt Totnam Lodge and redesigned the deer park, which at that date included long tree-lined walks and a deer "chase".The 4th Duke of Somerset bequeathed the unentailed Seymour estates to his niece Elizabeth Seymour, the wife of Thomas Bruce, 2nd Earl of Ailesbury (1656–1741), passing the Seymour estates to the Bruce family.In 1721 Elizabeth Seymour's son and heir, Charles Bruce, 3rd Earl of Ailesbury, rebuilt Totnam Lodge to the design of his brother-in-law the pioneering Palladian architect Lord Burlington.