 His father, William Fitzhugh, was a member of the Continental Congress.His sister, the noted Episcopalian lay leader Mary Lee Fitzhugh Custis, was the mother of Mary Anna Randolph Custis Lee.He was married to Anna Maria Sarah Goldsborough Fitzhugh.