'Give up' suggests that you have been doing something and you have decided to stop doing it. I gave up smoking years ago. People give up sugar because it is unhealthy. I give up travelling because i no longer enjoy it. 'Refuse' has no suggestion of hving done it before e,g I refuse to spend money on gambling. My child refused to go to bed last night without a teddy bear In thes examples the meaning for 'refuse' is simply to reject to perform an action, not to stop doing it as a habit.

"I won't give up travelling just because i have no money" if you want to use "refuse" you can say "I refuse to give up travelling just because I have no money" This 2nd example is a little stronger, maybe sounds like your a little angry/upset.