 He was the son of Robert Barker (1723–1780) and Sarah (née Folger) Gardner (1739–1833), who was born on Nantucket.His mother was a widow of Hezekiah Gardner, with whom she had a son, Gideon Gardner (1759–1832), who served as a U.S. Representative from Massachusetts.His parents married at Woods Hole, Massachusetts in 1723.