When I want to say "～ができなくて残念(ざんねん)でした” in Japanese, how would you say like that in English?

For example, "The other day I went to climb a mountain near my house, but my condition was not good ('I wasn't in great shape' is more natural). Because of that, when I was on my way to top of the mountain I gave up to climb. It was really disappointing ."

Is the last sentence natural? I looked up my dictionary, I found a lot of words meaning "ざんねん".
(pity, regret, sorrow, shame and so on)
But I didn't know which is the best word fitting my situation.

---

Your word choice is good here, I just made it a little more natural sounding.  'Unfortunate' or 'too bad' often works well for 残念 but here 'disappointed' is the best choice.

How would you say this ... in English?

Yes. You picked the best word. You really wanted to climb to the top, but you were disappointed that you weren't in condition to make it all the way. A few notes . . .

*"say something like that" - You can say "how would you say that in English" or "how would you say something like that in English." But you can't say "say like that."

** "gave up to climb" doesn't work. You wanted to say, "I gave up climbing." But a very natural phrase is "I gave up the climb." This way, you are treating "climb" like a noun instead of a verb. (Similarly, you can say, "I went for a climb" or in other situations, "I went for a swim" instead of "I went swimming" or "I took a walk" instead of "I went walking"). 

Regarding your question, "shame" is too strong. Shame means you were very embarrassed because you had done something wrong or had failed. "Sorrow" is very sad, and it usually involves a loss, or when someone is hurt. "Pity" either means かわいそう or なさけない. (Although you cant say, "It was a pity that I couldn't make it," But that doesn't sound right). "Regret" maybe. But disappointed or disappointment is the best for your situation.