As people have already said, you can learn how to speak the language without writing it.
But honestly, that seems a silly to me! It totally cuts out an entire communication method!
Kanji isn't nearly as hard as you think it is.
Firstly, you only have to learn about 1500-2000 kanji to be able to fully read a Japanese newspaper. Also keep in mind, that 300 of those 2000 kanji are used 70-80% of the time.
Yes, it takes time and patience, but it's not difficult to learn. There are a lot of other things about the language that are way harder.
In fact, I'm a lot better at reading and writing(Well...Typing. Lol) Japanese then I am at speaking it.
For example, just today my tutor said the word "かんどう” to me.
I was confused, I couldn't remember what "Kandou" meant.
But once she typed it out as 感動 (Emotionally moved), I instantly understood the word.
Personally I rely too much on kanji! I need to improve my listening and speaking skills a lot more.
But the point is, it's up to you to learn the language how you want.
But learning kanji isn't an impossible task. After you learn the first 100, you begin to see patterns and it starts becoming even easier to learn! Reading Japanese is really a wonderful thing, so I think it'd be a shame to not learn it. Especially when it helps with learning other aspects of the Japanese language.