There's a teaching that says, "As much as lies within you, be at peace with all men." It implicitly is indicating that there will be some who won't be at peace with you, no matter what you do, so you're on the right track by not letting it get to you.