 Around 1256, Aylmer de Lusignan obtained a licence to crenellate the 'insulam de Portand' and Robert, Earl of Gloucester, was granted a similar licence just 14 months later.It is generally presumed that Rufus castle is the site of any work that may have resulted from these licences and any remains that may date from the period exist only at foundation level, or have been lost to cliff erosion.The castle was rebuilt between 1432 and 1460, by Richard, Duke of York, and much of what remains today dates from this time.