 Doe began his education at Berwick Academy in Maine, and in 1844 Doe entered Phillips Exeter Academy.In 1845, Doe's father withdrew him from Exeter and enrolled him in Phillips Academy at Andover, which was less expensive.In autumn of 1845, Doe entered Harvard College where he stayed for only a year, possibly having been expelled for an incident in which he threw a tree stump through the dormitory window of an upperclassman who had tormented him in a hazing ritual.