in those days the great enemy , of whom sauron of mordor was but a servant , dwelt in angband in the north , and the elves of the west coming back to middle-earth made war upon him to regain the silmarils which he had stolen ; and the fathers of men aided the elves .