 He briefly married Elizabeth Mullen in 1759, and had a daughter, Harriet Luttrell.This marriage was later annulled.He married Jane Boyd in 1776, but they had no children and was succeeded by his brother John.