 Around the end of the 11th century William Rufus granted the estates in and around Kington to Adam Port and it is likely that he built the first castle here.One of his descendants, also called Adam Port, was in dispute with Henry II and as a consequence of this the castle was forfeited to the crown.In the late 12th century, William Braose held the castle for many years in his capacity as High Sheriff of Herefordshire.