 Olyphant was born on September 9, 1824 in Manhattan, New York City to David Olyphant and Ann Archer (McKenzie) Olyphant.After graduating from Columbia College in 1842, he entered his father's firm, Olyphant & Co., in 1842.Olyphant was named for the Scottish missionary Robert Morrison, who was a friend of his father from Canton (now Guangzhou), China.