Sometimes movies are hard to understand because the plot is not clear, so even native speakers don't quite understand what is going on at times... So I would not worry more than so much about that!
Casual conversation on the other hand is important. There are at least two factors involved:
The first is that there are English speakers from all over the world, with different accents, different local expressions, and sometimes also different vocabulary. You may not be understanding a particular accent or local manner of speech.
The other is that only by engaging in conversation, in which you actually interact with someone who uses the expressions, you get used to the expressions. Some students only want to talk during conversational sessions, but I believe that listening skills are only developed by having a true two-way conversation. So I recommend a lot of true conversation to resolve that problem. And always ask when you don't understand something - don't be at all ashamed to ask. People are usually happy to explain what they mean, and asking just shows your interest in the conversation.
If you need to understand a particular accent of English, it may be wise to choose your practice partners or your tutor accordingly.
Audio books with a written copy are good too, as you can mentally connect the sounds with the written words.
Enjoy your study : )