 Ellis was born in Rochester on October 17, 1852, the oldest of four sons of Dewitt and Eliza Haseltine Ellis.Childhood drawings suggest an unusual artistic aptitude.After public grade school and, for a while, a private high school academy in Rochester, Ellis entered the United States Military Academy at West Point in 1871 but was among the first-year cadets discharged after seven months for academic insufficiency, in his case in French and mathematics.