 In 1848 he married Ann Elizabeth McDowell of Wayne, New York, sister to Francis M. McDowell.Hallett and the McDowell brothers, along with Civil War General Nirom Crane, engaged in a number of enterprises including the Hallett & Co. Bank in New York City.In 1854 he built a large mansion in his wife's home town, known as the "Hallett House" or "The Aisle of Pines".