 Buffington was born in Little Valley, Cattaraugus County, New York in 1825, the son of Isaiah and Sophia (Winchester) Buffington.He was raised in Onondaga County, New York, until he was seventeen years old.He then came with his parents to Walworth County, Wisconsin, where he worked at farming, teaming, or anything by which he could make a living.