 According to the United States Census Bureau, the town has a total area of , of which is land and , or 4.06%, is water.Drummond is located southwest of the city of Ashland and northeast of the city of Hayward, within the Chequamegon-Nicolet National Forest.As of the census of 2000, there were 541 people, 231 households, and 155 families residing in the town.