The answer is likely that English speakers do not speak that fast, when compared to other native speakers speaking their native language. When listening to Japanese dialogue, I feel that they are communicating way faster than I could in English, but I believe that is only due to my lack of Japanese listening comprehension experience.

The answer is "practise practise practise" :) Anyone learning any language will think that everyone else is talking too quickly. This is not just for English, the same applies for learning any other language as well. People are not talking fast. It just seems that way at first.