charles 's involvement with the underground railroad had been the subject of quiet rumor within the family for some time now , but it was the sort of scandal that was seldom mentioned and never in the presence of anybody outside the hartman clan .