 The first castle on the site was built from earth and timber, with a keep and a motte protected by a palisade and a ditch.The Normans established a borough alongside the castle, which later became the village of Grosmont.The earl's landholdings in the region were slowly broken up after William fitz Osbern's son, Roger de Breteuil, rebelled against the Crown in 1075.