They weren't really staring at you.
He was led to believe that they were fixated on him, and he felt a sense of pride and joy, but his confidence was crushed when he was told that nobody there was staring at him, but instead staring at what was on him.