 Frotard, count of Ayssenès was seigneur of Broquiès around 1100, a scion of families of the Counts of Toulouse and of the Vicomtes d'Albi et Nîmes.He commenced building the castle and fortifications.In 1275 the seigneurship passed to the powerful Combret family who held it till it passed in marriage to the Arpajons in 1597.