Yes, your intuition is right. In polite conversation, we would simply say, "Sorry," if we cause a small, harmless inconvenience. If I need to talk to you, but you are listening to music, I might get your attention and casually say, "Sorry," to show that I know that I interrupted your music listening. If someone seriously offends someone else, as in your case, the offender should at least take the time to apologize with a full sentence: "Oh, I am so sorry. That was really insensitive of me." They should then pause what they are doing, make eye contact and ask if the other person is okay. In fact, sometimes, depending on the tone, "Sorry," can add insult to the original offense. Sometimes people simply say "Sorry," in a sarcastic tone. What they are really saying is, "Okay, you are way too sensitive. I'm saying 'sorry' only because I know you are expecting me to." I hope this person can approach you in more polite way and settle things, or maybe your don't want talk to him anymore. Anyway, best of luck.