Indeed, if what you do offends me but does not harm me otherwise, there should be a very high bar for prohibiting your act. After all, any ban, and certainly any vigilante acts to enforce it, may offend you as much, or more, than the offense to me. Excessive political correctness stifles progress as much as excessive license and disrespect.
Put differently, while you should avoid pressing the buttons that upset me to the extent possible, when you do push them you should explain carefully why that is necessary so as to move the debate forward, and how it should not be interpreted as a personal attack on me.