 After the death of the 3rd Earl, his son James, now 4th Earl, added the wings.The 4th Earl was deprived of his titles and estates after taking part in the Jacobite rising of 1715, although Panmure passed to his relatives, the Earls of Dalhousie.In 1852–1855, the house was extended on the instigation of Fox Maule, by the architect David Bryce in the Scottish Baronial style.