 He was the son of Sarah Gorgas and Philip Dorsheimer (1797–1868), a New York State Treasurer.He was educated in common schools, then at Phillips Andover Academy, and then studied at Harvard College from 1849 to 1851.He left Harvard without graduating because of a protracted illness.