 Ralph's son, William Lord Grey, was affirmed as the owner of the castle in 1625.The Grey dynasty maintained their ownership of the castle, which passed into Lady Mary Grey's side of the family following a law case in 1704.The lands around the castle and the outer bailey were used for growing wheat, barley and oats, and the walls were robbed of their stone for other building work.