 His father Ford Brown served as a purser in the Royal Navy, including a period serving under Sir Isaac Coffin and a period on HMS "Arethusa".He left the Navy after the end of the Napoleonic Wars.In 1818, Ford Brown married Caroline Madox, of an old Kentish family.