 On April 15, 1850 he married Mary Proctor Huger, the granddaughter of Daniel Elliott Huger.They had five children together.In 1856, he inherited a rice plantation in Georgetown County, South Carolina and moved there.