GO vs COME in a story

When talking to someone doesn't involve in the story, can 'come' still be a choice? Or is it the correct one?

"Every time we went and saw our grandfather in our childhood, he would ask the caterer for some dishes and treated us to them."

You personally never "come", you always "go" to some place. On the contrary, others never "go" to you, they always "come" to you. This is the easiest to way to remember. Hope it helps!