The problem is that the "western brain" is just not "wired" to efficiently memorize characters. We have evolved in the environment where all you need to know is 20+ letters that serve as the building building blocks for every single word you'll ever encounter as opposed to Chinese where there are 3,000+ of those unique building blocks.
Every single event of memorization, be it a new character, a word or a new experience creates a new neural connection in the brain, which is in essence a re-wiring process. While a child could relatively easily memorize a lot of things because his/her brain is still growing, for adults this process requires more time and efforts simply because our brain can only work with what is already available, therefore new neuron formation process is more difficult.
Anyway, this is my theory, it may not be strictly correct in biological sense but it makes sense to me.
I totally agree that learning characters is essential in mastering Chinese to a decent level, however there are millions of Chinese children that speak better Chinese than I do while hardly knowing any characters at all. That is really frustrating...