the quaint rhode island town was a center of commerce during colonial times and became a cultural retreat in the mid-nineteenth century with summer visitors that included henry wadsworth longfellow , henry and william james , and edgar allen poe .