 In c. 1820 his son enlarged the building, adding the second story and adding modest Federal styling.The building's exterior was replaced in the early 1980s in a historically sensitive manner.Ivory Perry is also credited with building a nearby schoolhouse and conducting school in this house for several years.