Actually, it is not as Eliot says. The speaker is saying that he does not mind, and that the other person's heart is not breaking. The speaker was not in love in the first place, so he doesn't care that the other person is not hurt. It is a complicated sentence.