none of the people of the valley dared to approach it , nor would they dwell near ; for they said that it was a trysting-place of the shadow-men , and there they would gather in times of fear , thronging round the stone and whispering .