 He is also the grandfather to Civil War general Robert E. Lee.George Washington mentions visiting the Lee House three times in his diaries.In 1825 the property was sold to Henry Fairfax, and later passed to John Fairfax in 1847.