Although there are vestiges of human presence in the castle of Castelo Branco dating back to the Pre and Proto-History, it was not until the Middle Ages that this Templar castle was built. Built by the Templars, probably between 1214 and 1230, it was delimited by an enclosure of walls and towers. King D. Dinis allegedly enlarged the limits of the castle, however there is evidence that it was King D. Afonso IV (1343) that erected its walls.
Through the engravings of the Fortress Book, written by Duarte de Armas, we are able to regard 16th century Castelo Branco as a fortress town with narrow streets and buildings with a wide door for the horse and a narrow door for the knight and his family.