Both are fine! I would naturally say "me." "my" strikes me as more formal, as an American, and but I remember reading that it's widely attested in UK speech. To add a "non-answer" on top of my answer, I would say that I don't often use that particular construction, and I think it would come across as awkwardly formal if I did. To communicate the same thing, I usually say, "Would you feel comfortable if I joined you?" or simply "would you mind if I joined you" (because asking if someone feels comfortable usually implies that they might not feel comfortable, and maybe you don't want to imply that).