the soldiers never did come back and dillon heard from the morrows that on february 17 , 1865 the city of columbia surrendered to general shermans army , and wade hampton 's confederate cavalry retreated from the city and by the end of february the city of charleston had fallen to the union and that it was shermans army that had passed bye the island plantation when cannon fire was heard .