the castle had been decimated at various times : by viking raiders from the north and west ; by normans from the eastthe last before king owains father had retaken anglesey from them over twenty years ago .