There is a difference between what is allowed by law and what is prudent in public. This is a double-edged sword. On one hand, as I related in an earlier post, where there are auditors/agitators/protestors present, returning with the purpose of confronting these individuals would not be in keeping with prudent behavior in regard to concealed carry.
If you are about your normal business and cannot avoid someone 'in your face', then move on. When you are not allowed to retreat, then--and only then--would it be prudent to exercise your rights. I don't think that is controversial.
In regard to grumpy grandma, if she's your responsibility, put an arm under hers and escort her, boy scout style, away for the situation. Retreating to higher ground, so to speak, is usually a winning strategy. You aren't carrying a weapon to keep from being a

, it is for turning those away that would do you and/or family bodily harm.
On that other hand, if you want to push the envelope of civil decency, understand there are those trying to live normal lives that are not the norm. They have security teams that follow their children. I am originally from the San Francisco Bay Area. On the mid-Peninsula, it would be safe to say there are a number of families in affluent neighborhoods that would not want their pictures taken for reason of personal safety. Imagine there will be many more as Middle East conflict escalates. Push a camera in a face that daddy does not want published on the internet and that camera becomes a distinct liability once you, the photographer, might be out of public view.
BTW, please note that karma is a Persian word.