"He came up to me (to the place where I was at the time). You could say "I went up to him (there, where he was)"

It depends on what you want to emphasize. If you want to emphasize departure, then "he went up to me" might work best. If you want to emphasize arrival, then "he came up to me" might work better. Context is also important. If he is on a lower floor of a building than you, then both of these clauses could sound natural. If you are both riding bikes and you are farther ahead, then "he caught up to me" would probably be more natural in North American English. I hope this helps! -Dustin