If I communicate, then it is OK. (HYPOTHESIS)

Right now, I am handling three languages; Malay language, Japanese language and English. After almost one year learning Japanese language, I can feel that my two main languages; Malay language and English are getting worst day by day. For Malay language, it is not so obvious because I am using it to communicate with my friends.
Unfortunately, my English is the worst language for me right now. I have one English class per week in my college. During the lesson, my English teacher gives many grammar exercises to strengthen the language. I realized something, even if I do as many exercises on the language as in only in writing, it still cannot help me in the language. The only right thing that I should do now is I have to communicate in English. Now, surprisingly I only speak English with my English teacher unlike the last five years, I speak English at school with my teachers and even my friends.

Same goes for to my Japanese language I think. We have four lessons of Japanese language per day, but we write and do grammar exercises in almost all lessons. Sometimes, we will have speaking lessons but I realize again that I am still not using the language to communicate. It is so difficult to communicate in other languages when everybody is using Malay language.

Now, I should start to think how to improve my three languages. Maybe the best way is just speak and do not care about what others may say. I guess?

Yes, just speak. You don't need to say language all the time. We know Japanese is a language.