 Chilton was born in Loudoun County, Virginia, to a prominent family.He was the son of William and Sarah (Powell) Chilton.He secured an appointment to the United States Military Academy and graduated 48th out of 57 cadets in the Class of 1837.