 His brother was Confederate colonel George S. Patton, Sr.

Patton graduated from the Virginia Military Institute in 1855, placing 2nd in a class of 16.While still a student, he became a member of the faculty, serving as acting assistant professor of Latin from 1852 until 1854.Patton also served as lieutenant in the cadet corps and for a time, as assistant professor of languages and assistant.