Personally, I think Cantonese (Chinese Dialect) is the most difficult.

The written and spoken languages structure are completely different. It has 7 (some argue it's 9) tones. It uses characters. And they don't really have a common Romanies system for learning. It's a tough one to learn.