You must have not been here. You must have never been here. Both sentences above are correct. The example you gave (you must never be here) is like a command and doesn't reflect whether the person has/has not done the action.

"You must have never been here" would be the grammatically correct version of what you're trying to say. Just take a look at your phrase "I HAVE never BEEN here" and follow the same grammar structure. I would say that even my corrected sentence is a little unnatural though, unless you are 100% sure that they haven't, so perhaps "I guess/assume you have never been here" would be better.