 Abbott married Margaret Anne Harriet in 1844 which produced a daughter Margaret but resulted in the death of his wife.He later married, Anna Matilda de Montmorency in 1868, however his second wife died shortly after having given birth to a son, James Reymond de Montmorency Abbott.The Pakistani city of Abbottabad as well as the district is named after him.