 In 1850 Chaffee married Eliza Irene Emerson (née Sanford) (1815 – 1903).Irene Emerson was the widow of Dr. John Emerson, the owner of the slave Dred Scott.She had a daughter, Henrietta Sanford King (Emerson) (1843 – 1919), from her first marriage.