 Because of negative publicity from the Scott case, Chaffee did not seek reelection in 1858 and became Librarian of the House of Representatives from 1860-1862.He then practiced medicine in Washington, D.C. until 1876, when he returned to Springfield.He died there in 1896 at age 84.