 When Elizabeth was three years of age, her father died of smallpox.As he had no sons, the title Earl Waldegrave passed to his brother, John.Elizabeth and her sisters subsequently took up residence with their mother at Ragman's Castle, a house in Twickenham.