He's calling you out, I realised.
I didn't think of it at first, I just thought it was friendly banter, but it soon dawned on me that he was trying to call you out in a way that shed you in a negative light to everyone around you.