 The castle then shortly passed to the Hopper family but was severely damaged in a fire which in 1796 destroyed most of the castle interior.In 1816 William Chaytor of Croft Hall, Yorkshire purchased the castle estate from the then owner, John Robert Hopper, for £78,000 and restored the fabric and rebuilt the interior in modern style.The estate was rich in coal and Witton Park Colliery was sunk in 1825.