When you see something is too expensive you say you cannot afford it, its the same way here, something is too expensive to lose.
Example: I cannot afford to give up on my dreams......which translates to the price of giving up my dreams is too high and cannot afford it.
Hope this helps!