@Chiaki, First of all my poor English can't delivery my thoughts.So I hope this makes sense. perhaps I couldn't give you a right input.
You are not alone. I steadily learned English for decades. If counting my primary school and vocational college in, I have learned English for more than 32 years. but I still feeling I am dumb and average level in English. I am not saying the exaggeration of my English. after I read
this book. You can download the script from
here. you will understand why as an Adult so difficult to learn a language. and what is the advantage of as an adult to learn a target language. you will improve your listening and vocab concurrently.
I think we cannot learn English like an educated native speaker does but I set my goal as phased achievement. I think the accurate and understandable English is good. good it is good enough that I consider myself as I have done with my English. because I am 42 years old and rising my kid is my priority. when talking about my kid. he is 19 months old. he started to mutter and stammer some whimsical sounds. he learns to speak the language, I don't think he is able to stammer accurate speech yet. I want to say we are not worse than a baby because we have consciousness.
After I read couple of comments here. I discover that many English learners reach fluid level that you can do that too. you are not exceptional one that you has same intelligence and smart brain.
I quite agree that when learning a new language you need to use it and practice it all the time. otherwise. if you are not relearning it. it will lose. so reinforcing is very important. so phased study is a good way to acquire new knowledge.
Japanese is not kind of roman language family. so your excuse is right. but it wasn't a barrier. if you understand the English essence you will find this language is easy to learn than many languages.
Thank you for reading, Have a nice day. Don't push yourself too hard but propel yourself a little progress.