It can be said in person or on the phone, but the implication is that the speaker is the person who wants to end the conversation without being impolite (ie ‘I’ve taken up your time long enough, I’ll let you go now’) - either because they a) genuinely think they’re now imposing, b) sense the listener wants the conversation to end or, more commonly, c) the speaker wants the conversation to end and wants to be polite about it (as the listener can then believe the speaker is only ending the conversation because the speaker doesn’t want to impose).