 Later alterations have included the purchase and removal of the fine paneling throughout the house to the Henry Francis du Pont Winterthur Museum.Charles Grahame, for whom the home is named, was associated with Frederick Calvert, sixth Lord Baltimore, through Grahame's brother, David Grahame (who married Calvert's cousin, Charlotte Hyde) and with Thomas Johnson, first elected Governor of the State of Maryland, through Grahame's son (who married Johnson's daughter).It was listed on the National Register of Historic Places in 1972.