Yes, those are both correct uses of "more." In each case, using the word "more" implies that what the first speaker said isn't totally wrong. In the first example ("Are you mad at me?" "No, I'm more surprised."), it sounds like person B is saying something like, "You're correct that I did have a strong emotional reaction, but it was surprise rather than anger." In the second example, it sounds like "Yes, it seems to be some kind of supernatural being, but I think it's a spirit rather than a ghost." By contrast, if you said "No, I'm not mad, I'm just surprised," then you seem to be making a slightly stronger distinction between anger and surprise. The meaning is very similar, though, and the two phrases would be interchangeable in most cases.

For the first question and answer, "more surprised" needs "...than mad" added at the end because it is part of a comparison (how they think you feel vs how you actually feel). "No, I'm more surprised than mad." (No, I feel this way instead of that way.) The second question/answer uses "more of a" correctly, with the phrase "...than a ghost" being implied and not needing to be said.

A: Are you mad at me? B: No, I'm more surprised. Is this clear that it means "No, I'm not mad at you. I'm just surprised."? No. Saying MORE implies that you are mad, but you are MORE surprised than mad. If you wish to say you're not mad, just surprised, all you have to do is omit the MORE

using 'more' is unnecessary in the context you've provided UNLESS you wish to say that "You're MORE surprised than Mad" Meaning, yes, you're a bit mad but you're more surprised. Else, just say 'No, I'm just surprised."