 In July 1781, Desha's family relocated to Fayette County, Kentucky, and the following year, they settled in what was then known as Cumberland district near the present-day city of Gallatin, Tennessee.Desha's younger brother, Robert, would later represent Tennessee in the U.S. House of Representatives.Like most frontier settlers, the Desha family frequently found themselves in conflict with American Indians after moving to Tennessee, and between the ages of 15 and 22, Joseph Desha volunteered in several military campaigns against them.