 Born in Newport, Rhode Island he was the son of Dr. David Olyphant, a Scottish supporter of Bonnie Prince Charlie, who arrived in America in the 1740s shortly after the Battle of Culloden, and Ann (née Vernon) Olyphant, granddaughter of Richard Ward, governor of Rhode Island.In 1806, following the death of his father, he went to New York where he worked for the firm of King and Talbot, who were involved in the China trade.After living in Baltimore between 18121817, he returned to New York where in 1820 he became an agent of Thomas H. Smith of Canton (now Guangzhou), China.