 Born in Saratoga Springs, New York, Chaffee graduated from the medical school of Middlebury College, Middlebury, Vermont, in 1835.He settled in Springfield, Massachusetts, where he began his medical practice.In 1854 he was elected on the American Party party ticket to the Thirty-fourth Congress as part of the Know Nothing party sweep of the Massachusetts congressional delegation that year.