 He married late, having had an illegitimate daughter, Elizabeth, by his first mistress, Mary Morgan, and marrying his second, the widow Sarah Lukin, on 13 February 1750 shortly before the birth of his son, William, on 3 May 1750.The year before, Windham's father Ashe died, and he inherited the family seat of Felbrigg Hall.Elizabeth went on to marry Thomas de Grey.