 It was inhabited until the Count's Feud in 1535, when it is speculated that it was destroyed.In 1577, the feudal tenant was granted permission to use the ruins as a quarry.Søborg Castle is first known from the 12th century, when ownership of the castle passed from the king to the Bishop of Roskilde.