I think it should be at the beginning.

Creo que lo mas dificil es la gramatica y entender los modismos

To speak is the hardest part

I think the hardest part of learning foreign language is the speaking and writing grammar in right way

For me, it has been trying to find information on the language (Slovak), especially if it is a lesser-known language. Some languages just promote themselves better I suppose - not related to any inherent difficulty in the language itself.