 As of December 2009, the town has a population of 3,756 inhabitants.Byzantine eparchial lists from the time of the 9th-10th century mention a bishop's centre by the name of Lyutitsa on the site of present-day Ivaylovgrad.According to John VI Kantakouzenos (1347–1354), his infantry reached the fortress in 1342–1343.