 Sir Ernest and Lady Margaret had two sons and four daughters.In 1890 their eldest son drowned at Tanjung Kling, Malacca, aged only seven years old.His other son, Patrick, followed his father's footstep and served in the Indian Civil Service.