this was the stronghold of saruman , as fame reported it ; for within living memory the men of rohan had not passed its gates , save perhaps a few , such as wormtongue , who came in secret and told no man what they saw .