If possible ask the person to speak slowly :) Yes it is difficult especially where there are so many accents among English speakers.

Well..as for this question,since our mother tongue is not English,I think there's no need for you to understand every word,you just need to figure out the context,the whole meaning of the dialogues and some important details...btw..can i know by what ways do you usually practice your sense of English listening?

That means you didn't practice enough listening to enable you to understand flluent conversations. You better just keep practicing ,whether listening,speaking or reading,it all will help improve your listening skills. You might at the moment be able to understand easy conversations, that you couldn't understand before,but not manage the more difficult fluent dialogues you hear. Don't worry,because that means you are improving ,but you have to exert more effort to meet the requirements of a higher level of comprehension. Good luck :)

I wouldn't think it as "a lot of" then if I were you.

Don't give up! Have a look at the tips in this italki site http://www.italki.com/knowledge/page/IMPROVE%20YOUR%20ENGLISH%20LISTENING%20SKILLS.htm