People can only annoy you as much as you let them.