 His eldest son died in infancy; the third son, William, saw action at Trafalgar on but drowned in 1806.There were also five daughters.He died at Tidworth in 1828, and the Tidworth and Vaynol estates were inherited by his namesake second son, Thomas Assheton Smith (1776–1858), who was also a noted amateur cricketer and all-round sportsman.