GO vs COME in a story

When talking to someone doesn't involve in the story, can 'come' still be a choice? Or is it the correct one?

"Every time we went and saw our grandfather in our childhood, he would ask the caterer for some dishes and treated us to them."

You personally never "come", you always "go" to some place. On the contrary, others never "go" to you, they always "come" to you. This is the easiest to way to remember. Hope it helps!

When talking to someone who isn't involved in the story, can 'come' still be a choice? Or is it the correct one? "Every time we [went] and saw our grandfather in our childhood, he would ask the caterer for some dishes and treated us to them." Any corrections on my errors welcome. Thank you! :)

In this case:

Came is for a location where you are/were

"My grandfather came to my house"

Went is for a location you are/were not (yet)

"I went to my grandfather's house"

If you were telling this story at your grandfather's house you would say "when we came here in our childhood" because that is the location where you are.

I hope this helps :)