The difference is the same as between "come" and "go".

Normally if you are the speaker then the location of the listener determines if you "come" or "go". ("back" in this case means a location you have previously visited.) For example, if you went to a friend's house, then left your wallet, you would call him and say you will "come back" to get it. He is home currently, and you will be arriving at his location. However, if you did the same thing but your friend left his house with you, you would instead tell him you need to "go back" to get it. He is not home and you will not be arriving at his location, but instead going from his current location back to where you were previously. If you leave work early, your boss might ask you to "come back" if he is at that location, or to "go back" if he is somewhere else.