Most worrying were the Seljuk Turks, who came out of the east in the 11th century to wrest large parts of Asia Minor from Constantinople's control.
Constantinople was ruling huge amounts of land, and so they did not have enough troops to protect them.