 The village is situated five kilometers west of the city of Tønsberg.Originally the municipality and the parish were named after the historic Sem Manor ("Sem hovedgård").During the Middle Ages, Sem Manor was a royal and feudal overlord residence at the site where Jarlsberg Manor is located today.