I don't know if this is the best idea, but try watching a lot of movies. It exposes you to different situations where you'll know how to respond certain statements. TED talk can also be a good practice, however I wouldn't really recommend watching TED talk alone. 

When we're learning a language, we tend to give our 100% focus to EVERY SINGLE WORD a native speaker would say, and when we miss a word or didn't catch its meaning, our minds tend to hold on to THAT word while the conversation is progressing. 

Try listening to people and understand the MEANING of what they're saying, and not EVERY WORD of what they're saying. Sooner or later, you'll realize that you've been subconsciously listening to what people say. Listening to the target language soon comes natural.

That's just my 2 cents. Hope it makes sense.