If something 'comes and goes' it means that sometimes it's there and sometimes it's not. I'd understand it to mean that there were certain periods when they were in the habit of speaking at 5 pm, and certain periods when they weren't. This has been going on for so long that it isn't even especially significant for the speaker.

I would need context but it seems to indicate that the author was in the habit of talking on the phone at 5pm with their friend. It has been so long since the they haev done this that they have almost forgot about it.