 It is built with Portland stone, with the walls roughly built of native ashlar.Rufus Castle features walls of roughly squared rubble and no roof.Three of the sides of the castle are considerably longer than the others.