"Afford' generally only has to do with money. It can also be used with time, "I cannot afford to waste any more time."

I can't afford you = ... 1. I don't have the money to hire you 2. I don't have the money to keep you (a girlfriend, a child)

If you can't tolerate somebody, you can say : I can't stand him! If you have no money to buy something, or you think that something it too expensive for you, you should use "can't afford": I can't afford this car. or I can't afford to go abroad.

The use of "can't afford" to mean "can't put up with", "can't tolerate" is quite an old-fashioned usage of the verb, in British English at least. It used to mean something along the lines of 'allow' or 'permit': "Could you afford me a light for my cigarette?" "He was afforded ten more days to finish his work by the Royal Society". However, this has fallen out of usage since the middle of the 20th century and nowadays, as Elena and Jim pointed out, "afford" is very much only to do with money nowadays. "I wanted a car, but I couldn't afford one."