 Business interests, mostly in whaling, banking and insurance, saw him resign from Government service in the late 1820s.Kelly's wife died in 1831 and two sons drowned in 1841 and 1842, respectively.He was financially ruined by the economic depression of the early 1840s and spent most of the remainder of his life back in the employ of the port authorities.