 Born in Londonderry, Rockingham County, New Hampshire, Patterson was the youngest of twelve children born to Thomas and Elizabeth (Wallace) Patterson.He received a common school education and graduated from Pinkerton Academy.At the age of 18 he taught school in New Hampshire for three months before moving to Livingston, New York with his older brother, where they ran a successful business dealing with the manufacture and sale of fanning mills.