 It was believed to have been a timber castle, the stronghold of the Wahull family, and later the de Grey family.In 1276, Walter Beywin is described as holding "7 selions 'above the castle of Rugemont'".The castle appears to have been attached to Brogborough Manor, where Oliver Cromwell is said to have stayed for a short time during the Civil War and to have made use of its entrenchments.