 Griffenfeldt named the farm Griffenfeldgård, but three years later it was renamed Jarlsberg Manor ("Jarlsberg Hovedgård").In 1682 the buildings on Jarlsberg burned and new buildings of stone were built by the new owner, the Danish-Norwegian Field Marshal Wilhelm Gustav Wedel.The municipality (originally the parish) is named after the old farm Sem (Norse "Sæmr" - a compressed form of "*Sæheimr"), since the first church was built there.