 In the middle of the 15th century the castle was described as being a "waste site with no annual value".The ruined castle and manor passed from the Baskerville family to the Vaughan family.Roger Vaughan was the son in law of Dafydd Gam and converted the castle and manor into a multi-gabled house.