Representation theory is a mathematical theory that describes the situation where elements of a group (here, the flavour rotations A in the group SU(3)) are automorphisms of a vector space (here, the set of all possible quantum states that you get from flavour-rotating a proton). Therefore, by studying the representation theory of SU(3), we can learn the possibilities for what the vector space is and how it is affected by flavour symmetry.